Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 08:10:39.539956 26028 test_util.cc:268] Using random seed: 94976703
I20240501 08:10:39.551395 26028 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:10:39.551481 26028 ts_itest-base.cc:113] --------------
I20240501 08:10:39.551523 26028 ts_itest-base.cc:114] 8 tablet servers
I20240501 08:10:39.551585 26028 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:10:39.551640 26028 ts_itest-base.cc:116] --------------
2024-05-01T08:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:10:39Z Disabled control of system clock
I20240501 08:10:39.575088 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.107.62:37731
--webserver_interface=127.25.107.62
--webserver_port=0
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.107.62:37731
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 08:10:39.893303 26040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:39.893630 26040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:39.893754 26040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:39.901196 26040 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:39.901340 26040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:10:39.901422 26040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:39.901502 26040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:10:39.901566 26040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:10:39.910493 26040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.25.107.62:37731
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.107.62:37731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.107.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:39.911772 26040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:39.913524 26040 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:39.923743 26048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.923799 26046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:39.923764 26045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:39.924152 26040 server_base.cc:1034] running on GCE node
I20240501 08:10:39.925089 26040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:39.926388 26040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:39.927592 26040 hybrid_clock.cc:648] HybridClock initialized: now 1714551039927535 us; error 45 us; skew 500 ppm
I20240501 08:10:39.927922 26040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:39.930027 26040 webserver.cc:460] Webserver started at http://127.25.107.62:37643/ using document root <none> and password file <none>
I20240501 08:10:39.930732 26040 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:39.930833 26040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:39.931149 26040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:39.933585 26040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2846a9ca19c842db9a3b834557ddb7bb"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-dv41"
I20240501 08:10:39.934365 26040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2846a9ca19c842db9a3b834557ddb7bb"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-dv41"
I20240501 08:10:39.939188 26040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240501 08:10:39.942397 26054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:39.943650 26040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:10:39.943807 26040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2846a9ca19c842db9a3b834557ddb7bb"
format_stamp: "Formatted at 2024-05-01 08:10:39 on dist-test-slave-dv41"
I20240501 08:10:39.943987 26040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:39.962102 26040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:39.962905 26040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:39.963166 26040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:39.980089 26040 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.62:37731
I20240501 08:10:39.980118 26105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.62:37731 every 8 connection(s)
I20240501 08:10:39.981421 26040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:10:39.985810 26106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:39.990418 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 26040
I20240501 08:10:39.990685 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:10:39.998241 26106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb: Bootstrap starting.
I20240501 08:10:40.001333 26106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:40.002290 26106 log.cc:824] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:40.004642 26106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb: No bootstrap required, opened a new log
I20240501 08:10:40.013079 26106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2846a9ca19c842db9a3b834557ddb7bb" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 37731 } }
I20240501 08:10:40.013491 26106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:40.013602 26106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2846a9ca19c842db9a3b834557ddb7bb, State: Initialized, Role: FOLLOWER
I20240501 08:10:40.014230 26106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2846a9ca19c842db9a3b834557ddb7bb" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 37731 } }
I20240501 08:10:40.014467 26106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:40.014602 26106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:40.014796 26106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:40.017542 26106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2846a9ca19c842db9a3b834557ddb7bb" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 37731 } }
I20240501 08:10:40.018150 26106 leader_election.cc:302] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2846a9ca19c842db9a3b834557ddb7bb; no voters:
I20240501 08:10:40.019078 26106 leader_election.cc:288] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:40.019196 26111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:40.020367 26111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [term 1 LEADER]: Becoming Leader. State: Replica: 2846a9ca19c842db9a3b834557ddb7bb, State: Running, Role: LEADER
I20240501 08:10:40.020922 26111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2846a9ca19c842db9a3b834557ddb7bb" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 37731 } }
I20240501 08:10:40.021354 26106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:40.023510 26113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2846a9ca19c842db9a3b834557ddb7bb. Latest consensus state: current_term: 1 leader_uuid: "2846a9ca19c842db9a3b834557ddb7bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2846a9ca19c842db9a3b834557ddb7bb" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 37731 } } }
I20240501 08:10:40.023823 26113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:40.024248 26112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2846a9ca19c842db9a3b834557ddb7bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2846a9ca19c842db9a3b834557ddb7bb" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 37731 } } }
I20240501 08:10:40.024506 26112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:40.024823 26117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:40.029897 26117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:40.038285 26117 catalog_manager.cc:1347] Generated new cluster ID: 9cce2be52135481ea03e70a6fb0acf2b
I20240501 08:10:40.038425 26117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:40.051247 26117 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:40.052937 26117 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:40.065425 26117 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2846a9ca19c842db9a3b834557ddb7bb: Generated new TSK 0
I20240501 08:10:40.066481 26117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:40.079684 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.1:0
--local_ip_for_outbound_sockets=127.25.107.1
--webserver_interface=127.25.107.1
--webserver_port=0
--tserver_master_addrs=127.25.107.62:37731
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 08:10:40.397899 26130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:40.398182 26130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:40.398313 26130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:40.405609 26130 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:40.405745 26130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:40.405920 26130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.1
I20240501 08:10:40.415216 26130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.107.1
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.1
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:40.416451 26130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:40.418126 26130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:40.429664 26136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.429760 26138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.429726 26135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:40.430557 26130 server_base.cc:1034] running on GCE node
I20240501 08:10:40.431118 26130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:40.431993 26130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:40.433178 26130 hybrid_clock.cc:648] HybridClock initialized: now 1714551040433128 us; error 45 us; skew 500 ppm
I20240501 08:10:40.433465 26130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:40.435634 26130 webserver.cc:460] Webserver started at http://127.25.107.1:37989/ using document root <none> and password file <none>
I20240501 08:10:40.436327 26130 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:40.436435 26130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:40.436733 26130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:40.439230 26130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c0624af4925d4686b5896f487f864943"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-dv41"
I20240501 08:10:40.439950 26130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c0624af4925d4686b5896f487f864943"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-dv41"
I20240501 08:10:40.444787 26130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:10:40.448105 26144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.449455 26130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 08:10:40.449609 26130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c0624af4925d4686b5896f487f864943"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-dv41"
I20240501 08:10:40.449841 26130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:40.477942 26130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:40.478866 26130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:40.479132 26130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:40.480090 26130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:40.481532 26130 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:40.481631 26130 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.481738 26130 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:40.481808 26130 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.511433 26130 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.1:40425
I20240501 08:10:40.511492 26256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.1:40425 every 8 connection(s)
I20240501 08:10:40.512732 26130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:10:40.515611 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 26130
I20240501 08:10:40.515828 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:10:40.519435 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.2:0
--local_ip_for_outbound_sockets=127.25.107.2
--webserver_interface=127.25.107.2
--webserver_port=0
--tserver_master_addrs=127.25.107.62:37731
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 08:10:40.526139 26257 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:40.526486 26257 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:40.527477 26257 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:40.529687 26071 ts_manager.cc:194] Registered new tserver with Master: c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:10:40.531196 26071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.1:49883
W20240501 08:10:40.838116 26260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:40.838425 26260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:40.838572 26260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:40.845845 26260 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:40.845960 26260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:40.846122 26260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.2
I20240501 08:10:40.855172 26260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.107.2
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.2
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:40.856424 26260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:40.858187 26260 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:40.868992 26266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.868957 26265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.869038 26268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:40.870009 26260 server_base.cc:1034] running on GCE node
I20240501 08:10:40.870530 26260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:40.871369 26260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:40.872556 26260 hybrid_clock.cc:648] HybridClock initialized: now 1714551040872520 us; error 31 us; skew 500 ppm
I20240501 08:10:40.872818 26260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:40.874845 26260 webserver.cc:460] Webserver started at http://127.25.107.2:37593/ using document root <none> and password file <none>
I20240501 08:10:40.875507 26260 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:40.875595 26260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:40.875859 26260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:40.878453 26260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "bc13665cb8894b1abf18738f527ab244"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-dv41"
I20240501 08:10:40.879140 26260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "bc13665cb8894b1abf18738f527ab244"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-dv41"
I20240501 08:10:40.883777 26260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 08:10:40.886902 26274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.888134 26260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:40.888275 26260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "bc13665cb8894b1abf18738f527ab244"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-dv41"
I20240501 08:10:40.888450 26260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:40.919600 26260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:40.920483 26260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:40.920735 26260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:40.921777 26260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:40.923280 26260 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:40.923372 26260 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.923475 26260 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:40.923542 26260 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.952657 26260 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.2:34319
I20240501 08:10:40.952720 26386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.2:34319 every 8 connection(s)
I20240501 08:10:40.954020 26260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:10:40.956655 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 26260
I20240501 08:10:40.956826 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:10:40.960125 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.3:0
--local_ip_for_outbound_sockets=127.25.107.3
--webserver_interface=127.25.107.3
--webserver_port=0
--tserver_master_addrs=127.25.107.62:37731
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 08:10:40.967222 26387 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:40.967561 26387 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:40.968487 26387 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:40.970011 26071 ts_manager.cc:194] Registered new tserver with Master: bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:40.971041 26071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.2:56581
W20240501 08:10:41.280035 26390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:41.280337 26390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:41.280452 26390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:41.288095 26390 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:41.288218 26390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:41.288347 26390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.3
I20240501 08:10:41.297346 26390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.107.3
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.3
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:41.298640 26390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:41.300264 26390 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:41.311223 26398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.311375 26396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.311411 26395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:41.312234 26390 server_base.cc:1034] running on GCE node
I20240501 08:10:41.312774 26390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:41.313617 26390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:41.314802 26390 hybrid_clock.cc:648] HybridClock initialized: now 1714551041314757 us; error 38 us; skew 500 ppm
I20240501 08:10:41.315105 26390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:41.325402 26390 webserver.cc:460] Webserver started at http://127.25.107.3:41195/ using document root <none> and password file <none>
I20240501 08:10:41.326110 26390 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:41.326226 26390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:41.326629 26390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:41.329087 26390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "875f11971b814f2ea7ce89de1e4f4587"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-dv41"
I20240501 08:10:41.329803 26390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "875f11971b814f2ea7ce89de1e4f4587"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-dv41"
I20240501 08:10:41.334705 26390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20240501 08:10:41.337962 26404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.339254 26390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:41.339407 26390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "875f11971b814f2ea7ce89de1e4f4587"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-dv41"
I20240501 08:10:41.339596 26390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:41.368039 26390 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:41.368863 26390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:41.369113 26390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:41.370010 26390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:41.371491 26390 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:41.371589 26390 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.371692 26390 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:41.371758 26390 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.401001 26390 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.3:41015
I20240501 08:10:41.401072 26516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.3:41015 every 8 connection(s)
I20240501 08:10:41.402304 26390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:10:41.408129 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 26390
I20240501 08:10:41.408326 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:10:41.411309 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.4:0
--local_ip_for_outbound_sockets=127.25.107.4
--webserver_interface=127.25.107.4
--webserver_port=0
--tserver_master_addrs=127.25.107.62:37731
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 08:10:41.414592 26517 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:41.414897 26517 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:41.415648 26517 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:41.416957 26071 ts_manager.cc:194] Registered new tserver with Master: 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:10:41.417833 26071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.3:51067
I20240501 08:10:41.534505 26257 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
W20240501 08:10:41.736505 26520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:41.736788 26520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:41.736950 26520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:41.744438 26520 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:41.744580 26520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:41.744746 26520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.4
I20240501 08:10:41.753836 26520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.107.4
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.4
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:41.755151 26520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:41.756955 26520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:41.768333 26528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.768364 26525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.768393 26526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:41.774605 26520 server_base.cc:1034] running on GCE node
I20240501 08:10:41.775213 26520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:41.776146 26520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:41.777336 26520 hybrid_clock.cc:648] HybridClock initialized: now 1714551041777287 us; error 47 us; skew 500 ppm
I20240501 08:10:41.777673 26520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:41.779912 26520 webserver.cc:460] Webserver started at http://127.25.107.4:37415/ using document root <none> and password file <none>
I20240501 08:10:41.780637 26520 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:41.780742 26520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:41.781040 26520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:41.783617 26520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f0f978a65b1549689c460fce4699e7a4"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-dv41"
I20240501 08:10:41.784341 26520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f0f978a65b1549689c460fce4699e7a4"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-dv41"
I20240501 08:10:41.789453 26520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:10:41.792615 26534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.794010 26520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:41.794145 26520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f0f978a65b1549689c460fce4699e7a4"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-dv41"
I20240501 08:10:41.794337 26520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:41.820657 26520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:41.821625 26520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:41.821889 26520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:41.822937 26520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:41.824419 26520 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:41.824502 26520 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.824584 26520 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:41.824637 26520 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.854598 26520 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.4:37061
I20240501 08:10:41.854657 26646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.4:37061 every 8 connection(s)
I20240501 08:10:41.855921 26520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:10:41.859377 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 26520
I20240501 08:10:41.859571 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:10:41.862756 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.5:0
--local_ip_for_outbound_sockets=127.25.107.5
--webserver_interface=127.25.107.5
--webserver_port=0
--tserver_master_addrs=127.25.107.62:37731
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 08:10:41.869246 26647 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:41.869565 26647 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:41.870532 26647 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:41.871982 26071 ts_manager.cc:194] Registered new tserver with Master: f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:10:41.872766 26071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.4:44711
I20240501 08:10:41.973996 26387 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
W20240501 08:10:42.191179 26650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:42.191486 26650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:42.191625 26650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:42.199466 26650 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:42.199609 26650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:42.199774 26650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.5
I20240501 08:10:42.208880 26650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.107.5
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.5
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:42.210192 26650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:42.211915 26650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:42.223618 26656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.223680 26658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.223655 26655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:42.224426 26650 server_base.cc:1034] running on GCE node
I20240501 08:10:42.225026 26650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:42.225961 26650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:42.227156 26650 hybrid_clock.cc:648] HybridClock initialized: now 1714551042227108 us; error 45 us; skew 500 ppm
I20240501 08:10:42.227509 26650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:42.229712 26650 webserver.cc:460] Webserver started at http://127.25.107.5:46711/ using document root <none> and password file <none>
I20240501 08:10:42.230461 26650 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:42.230571 26650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:42.230875 26650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:42.233409 26650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0c0ceab7acf1424aa23a96d23e0aa786"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-dv41"
I20240501 08:10:42.234171 26650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0c0ceab7acf1424aa23a96d23e0aa786"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-dv41"
I20240501 08:10:42.239516 26650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 08:10:42.242935 26664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.244361 26650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:42.244513 26650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0c0ceab7acf1424aa23a96d23e0aa786"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-dv41"
I20240501 08:10:42.244694 26650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:42.258142 26650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:42.259025 26650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:42.259284 26650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:42.260205 26650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:42.261628 26650 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:42.261721 26650 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.261826 26650 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:42.261893 26650 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.295325 26650 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.5:43333
I20240501 08:10:42.295380 26776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.5:43333 every 8 connection(s)
I20240501 08:10:42.296658 26650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:10:42.299381 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 26650
I20240501 08:10:42.299561 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:10:42.302446 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.6:0
--local_ip_for_outbound_sockets=127.25.107.6
--webserver_interface=127.25.107.6
--webserver_port=0
--tserver_master_addrs=127.25.107.62:37731
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 08:10:42.309572 26777 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:42.309877 26777 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:42.310745 26777 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:42.312232 26071 ts_manager.cc:194] Registered new tserver with Master: 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5:43333)
I20240501 08:10:42.312987 26071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.5:55361
I20240501 08:10:42.420230 26517 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
W20240501 08:10:42.631908 26780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:42.632216 26780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:42.632319 26780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:42.640017 26780 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:42.640153 26780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:42.640302 26780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.6
I20240501 08:10:42.649936 26780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.107.6
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.6
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:42.651298 26780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:42.653034 26780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:42.665299 26786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.665314 26788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.665334 26785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:42.666167 26780 server_base.cc:1034] running on GCE node
I20240501 08:10:42.666829 26780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:42.667843 26780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:42.669073 26780 hybrid_clock.cc:648] HybridClock initialized: now 1714551042669016 us; error 44 us; skew 500 ppm
I20240501 08:10:42.669415 26780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:42.671679 26780 webserver.cc:460] Webserver started at http://127.25.107.6:34541/ using document root <none> and password file <none>
I20240501 08:10:42.672425 26780 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:42.672535 26780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:42.672843 26780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:42.675583 26780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "f2a0e97dec014a17a4c96bfd2f60a96b"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-dv41"
I20240501 08:10:42.676378 26780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "f2a0e97dec014a17a4c96bfd2f60a96b"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-dv41"
I20240501 08:10:42.681850 26780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240501 08:10:42.685536 26794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.686954 26780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 08:10:42.687116 26780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "f2a0e97dec014a17a4c96bfd2f60a96b"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-dv41"
I20240501 08:10:42.687306 26780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:42.701388 26780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:42.702195 26780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:42.702508 26780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:42.703454 26780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:42.704891 26780 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:42.704984 26780 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.705152 26780 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:42.705222 26780 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.734102 26780 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.6:34341
I20240501 08:10:42.734154 26906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.6:34341 every 8 connection(s)
I20240501 08:10:42.735437 26780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:10:42.738999 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 26780
I20240501 08:10:42.739176 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:10:42.742462 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.7:0
--local_ip_for_outbound_sockets=127.25.107.7
--webserver_interface=127.25.107.7
--webserver_port=0
--tserver_master_addrs=127.25.107.62:37731
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 08:10:42.748574 26907 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:42.748898 26907 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:42.749835 26907 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:42.751308 26071 ts_manager.cc:194] Registered new tserver with Master: f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6:34341)
I20240501 08:10:42.752053 26071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.6:50691
I20240501 08:10:42.875301 26647 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
W20240501 08:10:43.061067 26910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:43.061345 26910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:43.061463 26910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:43.068819 26910 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:43.068975 26910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:43.069146 26910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.7
I20240501 08:10:43.078331 26910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.107.7
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.7
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:43.079667 26910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:43.081527 26910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:43.093389 26915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:43.093389 26918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:43.093837 26916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:43.094525 26910 server_base.cc:1034] running on GCE node
I20240501 08:10:43.095114 26910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:43.096021 26910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:43.097235 26910 hybrid_clock.cc:648] HybridClock initialized: now 1714551043097185 us; error 44 us; skew 500 ppm
I20240501 08:10:43.097544 26910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:43.099911 26910 webserver.cc:460] Webserver started at http://127.25.107.7:45449/ using document root <none> and password file <none>
I20240501 08:10:43.100641 26910 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:43.100750 26910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:43.101050 26910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:43.103680 26910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "fba0b24a97a84f6fa331664028231d19"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-dv41"
I20240501 08:10:43.104431 26910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "fba0b24a97a84f6fa331664028231d19"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-dv41"
I20240501 08:10:43.109545 26910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:10:43.112762 26924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.114007 26910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 08:10:43.114151 26910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fba0b24a97a84f6fa331664028231d19"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-dv41"
I20240501 08:10:43.114367 26910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:43.139719 26910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:43.140515 26910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:43.140761 26910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:43.141644 26910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:43.143127 26910 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:43.143216 26910 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.143317 26910 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:43.143383 26910 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.176782 26910 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.7:36815
I20240501 08:10:43.176833 27036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.7:36815 every 8 connection(s)
I20240501 08:10:43.177999 26910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:10:43.179672 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 26910
I20240501 08:10:43.179837 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:10:43.183063 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.8:0
--local_ip_for_outbound_sockets=127.25.107.8
--webserver_interface=127.25.107.8
--webserver_port=0
--tserver_master_addrs=127.25.107.62:37731
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 08:10:43.191167 27037 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:43.191431 27037 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:43.192234 27037 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:43.193516 26071 ts_manager.cc:194] Registered new tserver with Master: fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815)
I20240501 08:10:43.194247 26071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.7:41957
I20240501 08:10:43.315683 26777 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
W20240501 08:10:43.494953 27040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:43.495275 27040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:43.495424 27040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:43.502949 27040 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:43.503083 27040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:43.503234 27040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.8
I20240501 08:10:43.512987 27040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.25.107.8
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.8
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:43.514616 27040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:43.516407 27040 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:43.528518 27046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:43.528512 27048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:43.528558 27045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:43.529451 27040 server_base.cc:1034] running on GCE node
I20240501 08:10:43.530068 27040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:43.531029 27040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:43.532220 27040 hybrid_clock.cc:648] HybridClock initialized: now 1714551043532165 us; error 53 us; skew 500 ppm
I20240501 08:10:43.532513 27040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:43.534646 27040 webserver.cc:460] Webserver started at http://127.25.107.8:45177/ using document root <none> and password file <none>
I20240501 08:10:43.535418 27040 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:43.535529 27040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:43.535823 27040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:43.538437 27040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "25e18e28f3914e45be7093bf5bf436e5"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-dv41"
I20240501 08:10:43.539302 27040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "25e18e28f3914e45be7093bf5bf436e5"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-dv41"
I20240501 08:10:43.544412 27040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 08:10:43.547797 27054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.549333 27040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:43.549497 27040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "25e18e28f3914e45be7093bf5bf436e5"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-dv41"
I20240501 08:10:43.549695 27040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:43.568713 27040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:43.569658 27040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:43.569909 27040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:43.570932 27040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:43.572485 27040 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:43.572575 27040 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.572664 27040 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:43.572715 27040 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.604353 27040 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.8:33359
I20240501 08:10:43.604408 27166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.8:33359 every 8 connection(s)
I20240501 08:10:43.605783 27040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:10:43.610216 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 27040
I20240501 08:10:43.610463 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:10:43.619992 27167 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:43.620335 27167 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:43.621224 27167 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:43.622817 26071 ts_manager.cc:194] Registered new tserver with Master: 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8:33359)
I20240501 08:10:43.623626 26071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.8:56123
I20240501 08:10:43.628029 26028 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:10:43.650987 26028 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:10:43.651178 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 26650
I20240501 08:10:43.660413 26772 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:43.744688 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 26650
I20240501 08:10:43.755019 26907 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
I20240501 08:10:43.758913 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 26780
I20240501 08:10:43.767539 26902 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:43.848335 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 26780
I20240501 08:10:43.867272 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 26910
I20240501 08:10:43.875540 27032 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:43.958954 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 26910
I20240501 08:10:43.973275 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 27040
I20240501 08:10:43.982237 27162 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:44.068135 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 27040
I20240501 08:10:47.835208 26071 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34800:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:10:47.885025 26192 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc10e8d77e54d6f959552a962abee60 (DEFAULT_TABLE table=rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.888994 26581 tablet_service.cc:1467] Processing CreateTablet for tablet a5b85279a473435fb21f7e6c3cbebea2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.889385 26452 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc10e8d77e54d6f959552a962abee60 (DEFAULT_TABLE table=rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.890024 26451 tablet_service.cc:1467] Processing CreateTablet for tablet 595048c2a092461b979fe7e107882d75 (DEFAULT_TABLE table=rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.890889 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc10e8d77e54d6f959552a962abee60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.891470 26451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595048c2a092461b979fe7e107882d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.892297 26452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc10e8d77e54d6f959552a962abee60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.889803 26582 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc10e8d77e54d6f959552a962abee60 (DEFAULT_TABLE table=rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.894032 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc10e8d77e54d6f959552a962abee60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.894613 26191 tablet_service.cc:1467] Processing CreateTablet for tablet 595048c2a092461b979fe7e107882d75 (DEFAULT_TABLE table=rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.895088 26191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595048c2a092461b979fe7e107882d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.896909 26581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b85279a473435fb21f7e6c3cbebea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.898582 26190 tablet_service.cc:1467] Processing CreateTablet for tablet a5b85279a473435fb21f7e6c3cbebea2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.899042 26190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b85279a473435fb21f7e6c3cbebea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.905613 26322 tablet_service.cc:1467] Processing CreateTablet for tablet 595048c2a092461b979fe7e107882d75 (DEFAULT_TABLE table=rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.907313 26322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595048c2a092461b979fe7e107882d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.918910 26321 tablet_service.cc:1467] Processing CreateTablet for tablet a5b85279a473435fb21f7e6c3cbebea2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.919397 26321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b85279a473435fb21f7e6c3cbebea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.920732 27198 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:47.925871 27198 tablet_bootstrap.cc:654] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.926939 27197 tablet_bootstrap.cc:492] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:47.929811 27197 tablet_bootstrap.cc:654] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.931380 27197 log.cc:824] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:47.931380 27198 log.cc:824] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:47.932837 27196 tablet_bootstrap.cc:492] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:47.933504 27199 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:47.937247 27196 tablet_bootstrap.cc:654] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.937399 27199 tablet_bootstrap.cc:654] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.938112 27197 tablet_bootstrap.cc:492] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:47.938206 27198 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:47.938520 27197 ts_tablet_manager.cc:1367] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20240501 08:10:47.938570 27198 ts_tablet_manager.cc:1367] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240501 08:10:47.938630 27196 log.cc:824] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:47.938717 27199 log.cc:824] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:47.948532 27199 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:47.948561 27198 raft_consensus.cc:348] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:47.948575 27197 raft_consensus.cc:348] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:47.949069 27197 raft_consensus.cc:374] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.949156 27198 raft_consensus.cc:374] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.948974 27199 ts_tablet_manager.cc:1367] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20240501 08:10:47.949313 27198 raft_consensus.cc:729] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.949723 27197 raft_consensus.cc:729] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.950304 27198 consensus_queue.cc:260] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:47.950529 27197 consensus_queue.cc:260] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:47.952606 27196 tablet_bootstrap.cc:492] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:47.959980 27199 raft_consensus.cc:348] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:47.961385 27199 raft_consensus.cc:374] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.961580 27199 raft_consensus.cc:729] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.962452 27198 ts_tablet_manager.cc:1398] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.024s user 0.014s sys 0.009s
I20240501 08:10:47.962615 27199 consensus_queue.cc:260] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:47.963016 27198 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:47.965036 27198 tablet_bootstrap.cc:654] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.971364 27199 ts_tablet_manager.cc:1398] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.022s user 0.019s sys 0.000s
I20240501 08:10:47.972011 27199 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:47.977274 27196 ts_tablet_manager.cc:1367] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.004s
I20240501 08:10:47.978868 27199 tablet_bootstrap.cc:654] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.983127 27198 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:47.983381 27198 ts_tablet_manager.cc:1367] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.010s
I20240501 08:10:47.985019 27198 raft_consensus.cc:348] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:47.985441 27198 raft_consensus.cc:374] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.985589 27198 raft_consensus.cc:729] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.985890 27198 consensus_queue.cc:260] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:47.986531 27198 ts_tablet_manager.cc:1398] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:10:47.986934 27198 tablet_bootstrap.cc:492] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:47.988729 27198 tablet_bootstrap.cc:654] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:47.990810 27199 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:47.991050 27199 ts_tablet_manager.cc:1367] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.003s
I20240501 08:10:47.992379 27199 raft_consensus.cc:348] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:47.992866 27199 raft_consensus.cc:374] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:47.993027 27199 raft_consensus.cc:729] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:47.993233 27197 ts_tablet_manager.cc:1398] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.055s user 0.005s sys 0.034s
I20240501 08:10:47.993449 27199 consensus_queue.cc:260] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:47.996771 27199 ts_tablet_manager.cc:1398] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:10:47.998389 27197 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:48.000320 27197 tablet_bootstrap.cc:654] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.002578 27196 raft_consensus.cc:348] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.003199 27196 raft_consensus.cc:374] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.003357 27196 raft_consensus.cc:729] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.003435 27197 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:48.003664 27197 ts_tablet_manager.cc:1367] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240501 08:10:48.004577 27196 consensus_queue.cc:260] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.004860 27197 raft_consensus.cc:348] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.005059 27197 raft_consensus.cc:374] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.005160 27197 raft_consensus.cc:729] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.005388 27197 consensus_queue.cc:260] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.005874 27197 ts_tablet_manager.cc:1398] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:10:48.005932 27198 tablet_bootstrap.cc:492] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:48.006155 27198 ts_tablet_manager.cc:1367] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.013s
I20240501 08:10:48.007027 27198 raft_consensus.cc:348] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.007305 27198 raft_consensus.cc:374] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.007385 27198 raft_consensus.cc:729] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.007652 27198 consensus_queue.cc:260] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.008419 27198 ts_tablet_manager.cc:1398] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:10:48.010115 27196 ts_tablet_manager.cc:1398] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.033s user 0.016s sys 0.000s
I20240501 08:10:48.010602 27196 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:48.012102 27196 tablet_bootstrap.cc:654] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.014053 27196 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:48.014195 27196 ts_tablet_manager.cc:1367] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:10:48.015223 27196 raft_consensus.cc:348] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:48.015404 27196 raft_consensus.cc:374] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.015487 27196 raft_consensus.cc:729] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.015684 27196 consensus_queue.cc:260] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:48.016098 27196 ts_tablet_manager.cc:1398] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240501 08:10:48.020270 26258 tablet.cc:2369] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:10:48.112614 26648 tablet.cc:2369] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:48.117375 27204 raft_consensus.cc:482] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:48.117642 27204 raft_consensus.cc:504] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.119334 27204 leader_election.cc:288] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061), 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:10:48.126513 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc10e8d77e54d6f959552a962abee60" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4" is_pre_election: true
I20240501 08:10:48.126545 26472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc10e8d77e54d6f959552a962abee60" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587" is_pre_election: true
I20240501 08:10:48.127030 26472 raft_consensus.cc:2454] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 0.
I20240501 08:10:48.127034 26602 raft_consensus.cc:2454] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 0.
I20240501 08:10:48.127614 26148 leader_election.cc:302] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:48.127964 27204 raft_consensus.cc:2781] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:48.128123 27204 raft_consensus.cc:482] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:48.128226 27204 raft_consensus.cc:3037] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.130226 27204 raft_consensus.cc:504] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.130782 27211 raft_consensus.cc:482] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:48.130944 27204 leader_election.cc:288] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 election: Requested vote from peers f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061), 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:10:48.130949 27211 raft_consensus.cc:504] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:48.131335 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc10e8d77e54d6f959552a962abee60" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
I20240501 08:10:48.131510 26602 raft_consensus.cc:3037] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.132994 27211 leader_election.cc:288] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:10:48.133733 26472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc10e8d77e54d6f959552a962abee60" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
I20240501 08:10:48.133931 26472 raft_consensus.cc:3037] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.134670 26602 raft_consensus.cc:2454] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 1.
I20240501 08:10:48.136323 27204 raft_consensus.cc:482] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:48.136482 27204 raft_consensus.cc:504] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:48.136824 26472 raft_consensus.cc:2454] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 1.
I20240501 08:10:48.137293 27204 leader_election.cc:288] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:10:48.137352 26148 leader_election.cc:302] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:48.137581 27204 raft_consensus.cc:2781] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:48.138459 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b85279a473435fb21f7e6c3cbebea2" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4" is_pre_election: true
I20240501 08:10:48.138695 26602 raft_consensus.cc:2454] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 0.
I20240501 08:10:48.139060 26145 leader_election.cc:302] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0624af4925d4686b5896f487f864943, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:48.140081 27219 raft_consensus.cc:2781] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:48.140287 27219 raft_consensus.cc:482] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:48.140461 27219 raft_consensus.cc:3037] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.144583 27219 raft_consensus.cc:504] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:48.146095 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b85279a473435fb21f7e6c3cbebea2" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
I20240501 08:10:48.146288 27219 leader_election.cc:288] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 election: Requested vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:10:48.146401 26602 raft_consensus.cc:3037] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.146759 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b85279a473435fb21f7e6c3cbebea2" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:10:48.147369 26342 raft_consensus.cc:2454] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 0.
I20240501 08:10:48.148008 26535 leader_election.cc:302] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:48.148602 26602 raft_consensus.cc:2454] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 1.
I20240501 08:10:48.148730 27211 raft_consensus.cc:2741] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 08:10:48.149215 26145 leader_election.cc:302] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0624af4925d4686b5896f487f864943, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:48.149672 27219 raft_consensus.cc:2781] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:48.149886 27219 raft_consensus.cc:686] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 1 LEADER]: Becoming Leader. State: Replica: c0624af4925d4686b5896f487f864943, State: Running, Role: LEADER
I20240501 08:10:48.150522 27219 consensus_queue.cc:237] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:48.151654 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b85279a473435fb21f7e6c3cbebea2" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:10:48.151731 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b85279a473435fb21f7e6c3cbebea2" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:10:48.151916 26342 raft_consensus.cc:2454] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 0.
I20240501 08:10:48.154862 27204 raft_consensus.cc:686] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 1 LEADER]: Becoming Leader. State: Replica: c0624af4925d4686b5896f487f864943, State: Running, Role: LEADER
I20240501 08:10:48.155156 27204 consensus_queue.cc:237] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.157851 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b85279a473435fb21f7e6c3cbebea2" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943" is_pre_election: true
W20240501 08:10:48.159104 26518 tablet.cc:2369] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:48.159844 26070 catalog_manager.cc:5484] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0624af4925d4686b5896f487f864943 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "c0624af4925d4686b5896f487f864943" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:48.164642 26070 catalog_manager.cc:5484] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0624af4925d4686b5896f487f864943 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "c0624af4925d4686b5896f487f864943" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: UNKNOWN } } }
W20240501 08:10:48.211449 26388 tablet.cc:2369] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:48.244529 27205 raft_consensus.cc:482] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:48.244747 27205 raft_consensus.cc:504] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.246500 27205 leader_election.cc:288] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:10:48.253046 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595048c2a092461b979fe7e107882d75" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943" is_pre_election: true
I20240501 08:10:48.253255 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595048c2a092461b979fe7e107882d75" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:10:48.253361 26212 raft_consensus.cc:2454] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 0.
I20240501 08:10:48.253499 26341 raft_consensus.cc:2454] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 0.
I20240501 08:10:48.253942 26408 leader_election.cc:302] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:48.254392 27205 raft_consensus.cc:2781] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:48.254508 27205 raft_consensus.cc:482] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:48.254597 27205 raft_consensus.cc:3037] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.256618 27205 raft_consensus.cc:504] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.257308 27205 leader_election.cc:288] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 election: Requested vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:10:48.257666 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595048c2a092461b979fe7e107882d75" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:10:48.257783 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595048c2a092461b979fe7e107882d75" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943"
I20240501 08:10:48.257876 26341 raft_consensus.cc:3037] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.257993 26212 raft_consensus.cc:3037] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.259945 26341 raft_consensus.cc:2454] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
I20240501 08:10:48.259987 26212 raft_consensus.cc:2454] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
I20240501 08:10:48.260351 26405 leader_election.cc:302] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244; no voters:
I20240501 08:10:48.260644 27205 raft_consensus.cc:2781] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:48.261101 27205 raft_consensus.cc:686] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Becoming Leader. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Running, Role: LEADER
I20240501 08:10:48.261615 27205 consensus_queue.cc:237] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:48.266012 26070 catalog_manager.cc:5484] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: term changed from 0 to 1, leader changed from <none> to 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:48.542479 27219 consensus_queue.cc:1035] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:48.552433 27219 consensus_queue.cc:1035] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:48.558013 26341 raft_consensus.cc:3037] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.628964 27219 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:48.637650 27219 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:48.665822 27226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:48.666163 27226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:48.673772 27226 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 08:10:48.827320 26212 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:48.828207 26342 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:48.829617 27205 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:48.836479 27205 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:48.947712 27234 mvcc.cc:205] Tried to move back new op lower bound from 7022801095984812032 to 7022801093683679232. Current Snapshot: MvccSnapshot[applied={T|T < 7022801095984812032}]
W20240501 08:10:48.949599 26145 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:37061, user_credentials={real_user=slave}} blocked reactor thread for 46756us
INSERT report
rows total: 2000
time total: 276.68 ms
time per row: 0.13834 ms
I20240501 08:10:49.103260 26065 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34800:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:10:49.113191 26582 tablet_service.cc:1467] Processing CreateTablet for tablet 7206513860864277891e0f7159736271 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:49.113456 26452 tablet_service.cc:1467] Processing CreateTablet for tablet 7206513860864277891e0f7159736271 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:49.113760 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7206513860864277891e0f7159736271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.113915 26452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7206513860864277891e0f7159736271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.114055 26322 tablet_service.cc:1467] Processing CreateTablet for tablet 7206513860864277891e0f7159736271 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:49.114465 26322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7206513860864277891e0f7159736271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.115342 26450 tablet_service.cc:1467] Processing CreateTablet for tablet 697a56042bc14439b2ed2fa559e06a01 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:49.115816 26450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697a56042bc14439b2ed2fa559e06a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.118690 26321 tablet_service.cc:1467] Processing CreateTablet for tablet e97904ef0a5e41cab9054cbc84aae923 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:49.119103 26321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97904ef0a5e41cab9054cbc84aae923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.119103 26451 tablet_service.cc:1467] Processing CreateTablet for tablet e97904ef0a5e41cab9054cbc84aae923 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:49.119475 26451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97904ef0a5e41cab9054cbc84aae923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.123062 26192 tablet_service.cc:1467] Processing CreateTablet for tablet 697a56042bc14439b2ed2fa559e06a01 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:49.123518 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697a56042bc14439b2ed2fa559e06a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.126149 26581 tablet_service.cc:1467] Processing CreateTablet for tablet e97904ef0a5e41cab9054cbc84aae923 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:49.126643 27289 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:49.126679 26581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97904ef0a5e41cab9054cbc84aae923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.129253 27289 tablet_bootstrap.cc:654] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.129889 27288 tablet_bootstrap.cc:492] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:49.131351 26320 tablet_service.cc:1467] Processing CreateTablet for tablet 697a56042bc14439b2ed2fa559e06a01 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:49.131742 27288 tablet_bootstrap.cc:654] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.131906 26320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697a56042bc14439b2ed2fa559e06a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:49.140065 27289 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:49.140360 27289 ts_tablet_manager.cc:1367] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 08:10:49.141302 27289 raft_consensus.cc:348] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:49.141651 27287 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:49.141692 27289 raft_consensus.cc:374] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.141824 27288 tablet_bootstrap.cc:492] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:49.141911 27289 raft_consensus.cc:729] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.142298 27289 consensus_queue.cc:260] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:49.142616 27288 ts_tablet_manager.cc:1367] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 08:10:49.143172 27289 ts_tablet_manager.cc:1398] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:49.143445 27289 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:49.143738 27287 tablet_bootstrap.cc:654] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.144647 27288 raft_consensus.cc:348] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:49.145048 27288 raft_consensus.cc:374] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.145179 27289 tablet_bootstrap.cc:654] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.145203 27288 raft_consensus.cc:729] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.145591 27288 consensus_queue.cc:260] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:49.147167 27288 ts_tablet_manager.cc:1398] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:10:49.147547 27288 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:49.149515 27288 tablet_bootstrap.cc:654] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.153407 27289 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:49.153625 27289 ts_tablet_manager.cc:1367] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240501 08:10:49.154587 27289 raft_consensus.cc:348] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:49.154865 27289 raft_consensus.cc:374] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.154937 27289 raft_consensus.cc:729] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.155228 27289 consensus_queue.cc:260] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:49.155915 27289 ts_tablet_manager.cc:1398] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:10:49.156278 27289 tablet_bootstrap.cc:492] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:49.158100 27289 tablet_bootstrap.cc:654] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.159128 27288 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:49.159415 27288 ts_tablet_manager.cc:1367] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240501 08:10:49.160313 27288 raft_consensus.cc:348] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:49.160642 27289 tablet_bootstrap.cc:492] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:49.160799 27289 ts_tablet_manager.cc:1367] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:10:49.160653 27288 raft_consensus.cc:374] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.161048 27288 raft_consensus.cc:729] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.161568 27289 raft_consensus.cc:348] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:49.161768 27289 raft_consensus.cc:374] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.161855 27289 raft_consensus.cc:729] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.162031 27288 consensus_queue.cc:260] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:49.162111 27289 consensus_queue.cc:260] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:49.162791 27289 ts_tablet_manager.cc:1398] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:10:49.163311 27288 ts_tablet_manager.cc:1398] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:10:49.163683 27288 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:49.165489 27288 tablet_bootstrap.cc:654] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.169905 27292 tablet_bootstrap.cc:492] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:49.170697 27287 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:49.170882 27288 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:49.171020 27287 ts_tablet_manager.cc:1367] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.015s
I20240501 08:10:49.171067 27288 ts_tablet_manager.cc:1367] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:10:49.171844 27292 tablet_bootstrap.cc:654] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.171803 27288 raft_consensus.cc:348] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:49.171975 27288 raft_consensus.cc:374] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.172073 27288 raft_consensus.cc:729] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.172082 27287 raft_consensus.cc:348] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:49.172313 27287 raft_consensus.cc:374] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.172284 27288 consensus_queue.cc:260] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:49.172410 27287 raft_consensus.cc:729] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.172820 27288 ts_tablet_manager.cc:1398] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:49.173187 27287 consensus_queue.cc:260] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:49.174618 27287 ts_tablet_manager.cc:1398] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:49.174907 27287 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:49.175462 27292 tablet_bootstrap.cc:492] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:49.175647 27292 ts_tablet_manager.cc:1367] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 08:10:49.176653 27287 tablet_bootstrap.cc:654] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:49.176568 27292 raft_consensus.cc:348] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:49.176820 27292 raft_consensus.cc:374] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.176931 27292 raft_consensus.cc:729] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.177218 27292 consensus_queue.cc:260] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:49.177852 27292 ts_tablet_manager.cc:1398] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:10:49.179301 27287 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:49.179461 27287 ts_tablet_manager.cc:1367] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:10:49.180176 27287 raft_consensus.cc:348] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:49.180394 27287 raft_consensus.cc:374] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:49.180470 27287 raft_consensus.cc:729] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:49.180693 27287 consensus_queue.cc:260] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:49.181241 27287 ts_tablet_manager.cc:1398] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:10:49.181412 27299 raft_consensus.cc:482] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:49.181526 27299 raft_consensus.cc:504] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:49.183841 27299 leader_election.cc:288] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:49.184540 26340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7206513860864277891e0f7159736271" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:10:49.184736 26340 raft_consensus.cc:2454] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 0.
I20240501 08:10:49.185132 26535 leader_election.cc:302] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:49.185438 27299 raft_consensus.cc:2781] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:49.185554 27299 raft_consensus.cc:482] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:49.185634 27299 raft_consensus.cc:3037] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.189359 26472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7206513860864277891e0f7159736271" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587" is_pre_election: true
I20240501 08:10:49.189622 26472 raft_consensus.cc:2454] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 0.
I20240501 08:10:49.246809 27299 raft_consensus.cc:504] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:49.247639 27299 leader_election.cc:288] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 election: Requested vote from peers 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:49.248225 26472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7206513860864277891e0f7159736271" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
I20240501 08:10:49.248270 26340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7206513860864277891e0f7159736271" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:10:49.248463 26472 raft_consensus.cc:3037] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.248515 26340 raft_consensus.cc:3037] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.250551 26472 raft_consensus.cc:2454] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 1.
I20240501 08:10:49.251082 26538 leader_election.cc:302] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:49.251432 27299 raft_consensus.cc:2781] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:49.251468 26340 raft_consensus.cc:2454] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 1.
I20240501 08:10:49.251616 27299 raft_consensus.cc:686] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Becoming Leader. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Running, Role: LEADER
I20240501 08:10:49.252327 27299 consensus_queue.cc:237] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:49.256189 26065 catalog_manager.cc:5484] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0f978a65b1549689c460fce4699e7a4 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:49.261852 27277 raft_consensus.cc:482] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:49.262035 27277 raft_consensus.cc:504] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:49.262825 27277 leader_election.cc:288] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:10:49.263442 26340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a56042bc14439b2ed2fa559e06a01" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:10:49.263437 26472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a56042bc14439b2ed2fa559e06a01" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587" is_pre_election: true
I20240501 08:10:49.263674 26472 raft_consensus.cc:2454] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 0.
I20240501 08:10:49.263693 26340 raft_consensus.cc:2454] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 0.
I20240501 08:10:49.264149 26148 leader_election.cc:302] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:49.264432 27277 raft_consensus.cc:2781] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:49.264554 27277 raft_consensus.cc:482] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:49.264626 27277 raft_consensus.cc:3037] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.305454 27205 raft_consensus.cc:482] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:49.305680 27205 raft_consensus.cc:504] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:49.306481 27205 leader_election.cc:288] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:10:49.307159 26340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97904ef0a5e41cab9054cbc84aae923" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:10:49.307570 26340 raft_consensus.cc:2454] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 0.
I20240501 08:10:49.307996 26405 leader_election.cc:302] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244; no voters:
I20240501 08:10:49.308310 27205 raft_consensus.cc:2781] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:49.308429 27205 raft_consensus.cc:482] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:49.308545 27205 raft_consensus.cc:3037] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.313254 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97904ef0a5e41cab9054cbc84aae923" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4" is_pre_election: true
I20240501 08:10:49.313510 26602 raft_consensus.cc:2454] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 0.
I20240501 08:10:49.355623 27205 raft_consensus.cc:504] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:49.355669 27277 raft_consensus.cc:504] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:49.356413 27205 leader_election.cc:288] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 election: Requested vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:10:49.357014 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97904ef0a5e41cab9054cbc84aae923" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:10:49.357100 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97904ef0a5e41cab9054cbc84aae923" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
I20240501 08:10:49.357281 26602 raft_consensus.cc:3037] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.357291 26342 raft_consensus.cc:3037] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.357479 26472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a56042bc14439b2ed2fa559e06a01" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
I20240501 08:10:49.357681 26472 raft_consensus.cc:3037] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.359843 26340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a56042bc14439b2ed2fa559e06a01" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:10:49.360042 27277 leader_election.cc:288] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 election: Requested vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:10:49.360065 26340 raft_consensus.cc:3037] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:49.448843 26602 raft_consensus.cc:2454] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
I20240501 08:10:49.448836 26342 raft_consensus.cc:2454] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
I20240501 08:10:49.448843 26472 raft_consensus.cc:2454] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 1.
I20240501 08:10:49.449618 26405 leader_election.cc:302] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244; no voters:
I20240501 08:10:49.449623 26148 leader_election.cc:302] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:49.449987 27277 raft_consensus.cc:2781] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:49.450322 27205 raft_consensus.cc:2781] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:49.450466 27277 raft_consensus.cc:686] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 1 LEADER]: Becoming Leader. State: Replica: c0624af4925d4686b5896f487f864943, State: Running, Role: LEADER
I20240501 08:10:49.450662 27205 raft_consensus.cc:686] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Becoming Leader. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Running, Role: LEADER
I20240501 08:10:49.450790 27277 consensus_queue.cc:237] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:49.450978 27205 consensus_queue.cc:237] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:49.451853 26340 raft_consensus.cc:2454] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 1.
I20240501 08:10:49.454664 26065 catalog_manager.cc:5484] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0624af4925d4686b5896f487f864943 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "c0624af4925d4686b5896f487f864943" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:49.454684 26066 catalog_manager.cc:5484] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: term changed from 0 to 1, leader changed from <none> to 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:49.760756 27310 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:49.769577 27313 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:49.865146 27309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:49.865552 27309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:49.873462 27309 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:10:49.962581 27277 consensus_queue.cc:1035] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:50.001436 27277 consensus_queue.cc:1035] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:50.016392 27205 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:50.066032 26602 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:10:50.070590 27205 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:50.139519 27361 mvcc.cc:205] Tried to move back new op lower bound from 7022801100993486848 to 7022801098554773504. Current Snapshot: MvccSnapshot[applied={T|T < 7022801100993486848}]
INSERT report
rows total: 2000
time total: 251.83 ms
time per row: 0.125915 ms
I20240501 08:10:50.276063 26069 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34800:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:10:50.285230 26192 tablet_service.cc:1467] Processing CreateTablet for tablet 12641e2d23ac450f9f04e41bb583b32d (DEFAULT_TABLE table=rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:50.285380 26320 tablet_service.cc:1467] Processing CreateTablet for tablet 12641e2d23ac450f9f04e41bb583b32d (DEFAULT_TABLE table=rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:50.285719 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12641e2d23ac450f9f04e41bb583b32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:50.285820 26320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12641e2d23ac450f9f04e41bb583b32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:50.285990 26191 tablet_service.cc:1467] Processing CreateTablet for tablet d6fff31fdd6b496198695328538e11f0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:50.286396 26191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fff31fdd6b496198695328538e11f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:50.286754 26451 tablet_service.cc:1467] Processing CreateTablet for tablet d6fff31fdd6b496198695328538e11f0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:50.287149 26451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fff31fdd6b496198695328538e11f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:50.287598 26582 tablet_service.cc:1467] Processing CreateTablet for tablet 12641e2d23ac450f9f04e41bb583b32d (DEFAULT_TABLE table=rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:50.288009 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12641e2d23ac450f9f04e41bb583b32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:50.292239 26190 tablet_service.cc:1467] Processing CreateTablet for tablet f01a4fd9d73849ac8ac01c7a6470fb2c (DEFAULT_TABLE table=rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:50.292603 26190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01a4fd9d73849ac8ac01c7a6470fb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:50.294463 26581 tablet_service.cc:1467] Processing CreateTablet for tablet d6fff31fdd6b496198695328538e11f0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:50.294903 26581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fff31fdd6b496198695328538e11f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:50.298280 27377 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:50.298431 27378 tablet_bootstrap.cc:492] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:50.298741 26580 tablet_service.cc:1467] Processing CreateTablet for tablet f01a4fd9d73849ac8ac01c7a6470fb2c (DEFAULT_TABLE table=rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:50.299201 26580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01a4fd9d73849ac8ac01c7a6470fb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:50.301129 27378 tablet_bootstrap.cc:654] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:50.301949 27377 tablet_bootstrap.cc:654] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:50.303290 26322 tablet_service.cc:1467] Processing CreateTablet for tablet f01a4fd9d73849ac8ac01c7a6470fb2c (DEFAULT_TABLE table=rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:50.303781 26322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01a4fd9d73849ac8ac01c7a6470fb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:50.311450 27376 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:50.312608 27377 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:50.312853 27377 ts_tablet_manager.cc:1367] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20240501 08:10:50.313580 27376 tablet_bootstrap.cc:654] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:50.313717 27377 raft_consensus.cc:348] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:50.314054 27377 raft_consensus.cc:374] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:50.314213 27377 raft_consensus.cc:729] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:50.314610 27377 consensus_queue.cc:260] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:50.317253 27381 tablet_bootstrap.cc:492] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:50.319402 27381 tablet_bootstrap.cc:654] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:50.319617 27377 ts_tablet_manager.cc:1398] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:10:50.320508 27377 tablet_bootstrap.cc:492] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:50.320693 27378 tablet_bootstrap.cc:492] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:50.320914 27378 ts_tablet_manager.cc:1367] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.012s
I20240501 08:10:50.321911 27378 raft_consensus.cc:348] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:50.322156 27378 raft_consensus.cc:374] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:50.322208 27377 tablet_bootstrap.cc:654] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:50.322327 27378 raft_consensus.cc:729] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:50.322607 27378 consensus_queue.cc:260] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:50.323310 27378 ts_tablet_manager.cc:1398] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:10:50.323661 27378 tablet_bootstrap.cc:492] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:50.325351 27378 tablet_bootstrap.cc:654] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:50.326408 27376 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:50.327813 27376 ts_tablet_manager.cc:1367] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20240501 08:10:50.328842 27376 raft_consensus.cc:348] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:50.329093 27376 raft_consensus.cc:374] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:50.329231 27376 raft_consensus.cc:729] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:50.329510 27376 consensus_queue.cc:260] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:50.330082 27376 ts_tablet_manager.cc:1398] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:50.330461 27376 tablet_bootstrap.cc:492] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:50.330686 27378 tablet_bootstrap.cc:492] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:50.330900 27378 ts_tablet_manager.cc:1367] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240501 08:10:50.331717 27378 raft_consensus.cc:348] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:50.332001 27378 raft_consensus.cc:374] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:50.332051 27376 tablet_bootstrap.cc:654] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:50.332150 27378 raft_consensus.cc:729] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:50.332477 27378 consensus_queue.cc:260] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:50.333209 27378 ts_tablet_manager.cc:1398] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:50.333560 27378 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:50.335240 27378 tablet_bootstrap.cc:654] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:50.337020 27377 tablet_bootstrap.cc:492] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:50.337260 27377 ts_tablet_manager.cc:1367] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20240501 08:10:50.337538 27376 tablet_bootstrap.cc:492] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:50.337728 27376 ts_tablet_manager.cc:1367] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:10:50.337934 27377 raft_consensus.cc:348] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:50.338136 27377 raft_consensus.cc:374] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:50.338207 27377 raft_consensus.cc:729] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:50.338487 27377 consensus_queue.cc:260] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:50.338570 27376 raft_consensus.cc:348] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:50.338886 27376 raft_consensus.cc:374] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:50.339000 27376 raft_consensus.cc:729] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:50.339094 27377 ts_tablet_manager.cc:1398] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:50.339437 27377 tablet_bootstrap.cc:492] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:50.339327 27376 consensus_queue.cc:260] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:50.341149 27376 ts_tablet_manager.cc:1398] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:50.341773 27377 tablet_bootstrap.cc:654] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:50.342440 27381 tablet_bootstrap.cc:492] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:50.342684 27381 ts_tablet_manager.cc:1367] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.006s
I20240501 08:10:50.343771 27381 raft_consensus.cc:348] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:50.344107 27381 raft_consensus.cc:374] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:50.344274 27377 tablet_bootstrap.cc:492] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:50.344434 27377 ts_tablet_manager.cc:1367] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:10:50.344996 27378 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:50.345134 27378 ts_tablet_manager.cc:1367] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 08:10:50.345192 27377 raft_consensus.cc:348] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:50.345407 27377 raft_consensus.cc:374] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:50.345508 27377 raft_consensus.cc:729] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:50.345739 27381 raft_consensus.cc:729] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:50.345743 27377 consensus_queue.cc:260] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:50.346030 27381 consensus_queue.cc:260] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:50.346315 27377 ts_tablet_manager.cc:1398] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:10:50.345818 27378 raft_consensus.cc:348] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:50.346720 27378 raft_consensus.cc:374] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:50.346866 27378 raft_consensus.cc:729] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:50.346892 27381 ts_tablet_manager.cc:1398] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:10:50.347153 27378 consensus_queue.cc:260] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:50.347891 27378 ts_tablet_manager.cc:1398] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:50.365016 27313 raft_consensus.cc:482] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:50.365365 27313 raft_consensus.cc:504] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:50.366034 27313 leader_election.cc:288] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:50.366533 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01a4fd9d73849ac8ac01c7a6470fb2c" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943" is_pre_election: true
I20240501 08:10:50.366580 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01a4fd9d73849ac8ac01c7a6470fb2c" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:10:50.366760 26212 raft_consensus.cc:2454] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 0.
I20240501 08:10:50.366801 26341 raft_consensus.cc:2454] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 0.
I20240501 08:10:50.367179 26538 leader_election.cc:302] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0624af4925d4686b5896f487f864943, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:50.367483 27313 raft_consensus.cc:2781] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:50.367579 27313 raft_consensus.cc:482] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:50.367682 27313 raft_consensus.cc:3037] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:50.454972 27313 raft_consensus.cc:504] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:50.455736 27313 leader_election.cc:288] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 election: Requested vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:50.456282 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01a4fd9d73849ac8ac01c7a6470fb2c" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943"
I20240501 08:10:50.456394 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01a4fd9d73849ac8ac01c7a6470fb2c" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:10:50.456506 26212 raft_consensus.cc:3037] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:50.456661 26341 raft_consensus.cc:3037] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:50.534818 27313 raft_consensus.cc:482] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:50.535012 27313 raft_consensus.cc:504] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:50.535658 27313 leader_election.cc:288] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:50.536219 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:10:50.536211 26211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943" is_pre_election: true
I20240501 08:10:50.536443 26342 raft_consensus.cc:2454] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 0.
I20240501 08:10:50.536458 26211 raft_consensus.cc:2454] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 0.
I20240501 08:10:50.536865 26535 leader_election.cc:302] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, c0624af4925d4686b5896f487f864943, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:50.537185 27313 raft_consensus.cc:2781] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:50.537281 27313 raft_consensus.cc:482] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:50.537393 27313 raft_consensus.cc:3037] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:50.557087 26341 raft_consensus.cc:2454] T f01a4fd9d73849ac8ac01c7a6470fb2c P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 1.
I20240501 08:10:50.557088 26212 raft_consensus.cc:2454] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 1.
I20240501 08:10:50.557106 27313 raft_consensus.cc:504] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:50.557855 26535 leader_election.cc:302] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:50.557946 27313 leader_election.cc:288] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 election: Requested vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:50.558238 27313 raft_consensus.cc:2781] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:50.558498 27313 raft_consensus.cc:686] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Becoming Leader. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Running, Role: LEADER
I20240501 08:10:50.558768 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:10:50.558758 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943"
I20240501 08:10:50.558988 26212 raft_consensus.cc:3037] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:50.559022 26341 raft_consensus.cc:3037] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:50.558815 27313 consensus_queue.cc:237] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:50.563704 26067 catalog_manager.cc:5484] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0f978a65b1549689c460fce4699e7a4 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:50.636498 27313 raft_consensus.cc:482] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:50.636696 27313 raft_consensus.cc:504] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:50.637378 27313 leader_election.cc:288] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:10:50.638056 26211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fff31fdd6b496198695328538e11f0" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943" is_pre_election: true
I20240501 08:10:50.638094 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fff31fdd6b496198695328538e11f0" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587" is_pre_election: true
I20240501 08:10:50.638355 26471 raft_consensus.cc:2454] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 0.
I20240501 08:10:50.638355 26211 raft_consensus.cc:2454] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 0.
I20240501 08:10:50.638829 26538 leader_election.cc:302] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:50.639135 27313 raft_consensus.cc:2781] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:50.639256 27313 raft_consensus.cc:482] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:50.639348 27313 raft_consensus.cc:3037] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:50.654444 26212 raft_consensus.cc:2454] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 1.
I20240501 08:10:50.654444 26341 raft_consensus.cc:2454] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 1.
I20240501 08:10:50.654467 27313 raft_consensus.cc:504] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:50.655138 26535 leader_election.cc:302] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:50.655411 27313 leader_election.cc:288] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 election: Requested vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:10:50.655526 27310 raft_consensus.cc:2781] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:50.655714 27310 raft_consensus.cc:686] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Becoming Leader. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Running, Role: LEADER
I20240501 08:10:50.656013 27310 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:50.656210 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fff31fdd6b496198695328538e11f0" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943"
I20240501 08:10:50.656421 26212 raft_consensus.cc:3037] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:50.656550 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fff31fdd6b496198695328538e11f0" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
I20240501 08:10:50.656759 26471 raft_consensus.cc:3037] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:50.660194 26067 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0f978a65b1549689c460fce4699e7a4 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:50.753499 26471 raft_consensus.cc:2454] T d6fff31fdd6b496198695328538e11f0 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 1.
I20240501 08:10:50.753517 26212 raft_consensus.cc:2454] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 1.
I20240501 08:10:50.754091 26538 leader_election.cc:302] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:50.754519 27310 raft_consensus.cc:2781] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:50.754719 27310 raft_consensus.cc:686] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Becoming Leader. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Running, Role: LEADER
I20240501 08:10:50.755089 27310 consensus_queue.cc:237] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:50.757962 26067 catalog_manager.cc:5484] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0f978a65b1549689c460fce4699e7a4 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:51.047289 27310 consensus_queue.cc:1035] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:51.056283 27310 consensus_queue.cc:1035] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:51.227932 27392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:51.228281 27392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:10:51.233517 27310 consensus_queue.cc:1035] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:51.236822 27392 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:10:51.239485 27313 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:51.241966 27313 consensus_queue.cc:1035] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:51.245885 27396 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
W20240501 08:10:51.461673 26538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.1:40425, user_credentials={real_user=slave}} blocked reactor thread for 64755.5us
INSERT report
rows total: 2000
time total: 174.393 ms
time per row: 0.0871964 ms
I20240501 08:10:51.582113 26067 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34800:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:10:51.592475 26320 tablet_service.cc:1467] Processing CreateTablet for tablet 9981240ef4c1466cb37895e24a0c6e5c (DEFAULT_TABLE table=rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.592926 26451 tablet_service.cc:1467] Processing CreateTablet for tablet 9981240ef4c1466cb37895e24a0c6e5c (DEFAULT_TABLE table=rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.593035 26320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981240ef4c1466cb37895e24a0c6e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.592926 26192 tablet_service.cc:1467] Processing CreateTablet for tablet 9981240ef4c1466cb37895e24a0c6e5c (DEFAULT_TABLE table=rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.593351 26451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981240ef4c1466cb37895e24a0c6e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.593372 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981240ef4c1466cb37895e24a0c6e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.594156 26452 tablet_service.cc:1467] Processing CreateTablet for tablet 99c45caee3514df08b819cf61954f88d (DEFAULT_TABLE table=rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.594590 26452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c45caee3514df08b819cf61954f88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.599303 26190 tablet_service.cc:1467] Processing CreateTablet for tablet 99c45caee3514df08b819cf61954f88d (DEFAULT_TABLE table=rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.599316 26582 tablet_service.cc:1467] Processing CreateTablet for tablet 99c45caee3514df08b819cf61954f88d (DEFAULT_TABLE table=rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.599790 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c45caee3514df08b819cf61954f88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.599798 26190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c45caee3514df08b819cf61954f88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.601346 26322 tablet_service.cc:1467] Processing CreateTablet for tablet c3ddc36d3fc146938c366df2c7f090a6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.601873 26322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ddc36d3fc146938c366df2c7f090a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.603222 26450 tablet_service.cc:1467] Processing CreateTablet for tablet c3ddc36d3fc146938c366df2c7f090a6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.603796 26450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ddc36d3fc146938c366df2c7f090a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.604885 27451 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:51.606734 26580 tablet_service.cc:1467] Processing CreateTablet for tablet c3ddc36d3fc146938c366df2c7f090a6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:51.607228 26580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ddc36d3fc146938c366df2c7f090a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.609499 27451 tablet_bootstrap.cc:654] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.615509 27454 tablet_bootstrap.cc:492] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:51.618957 27454 tablet_bootstrap.cc:654] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.620267 27451 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:51.621038 27451 ts_tablet_manager.cc:1367] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240501 08:10:51.621369 27455 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:51.621942 27452 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:51.622534 27451 raft_consensus.cc:348] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.622927 27451 raft_consensus.cc:374] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.623046 27451 raft_consensus.cc:729] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.623410 27451 consensus_queue.cc:260] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.624038 27452 tablet_bootstrap.cc:654] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.624280 27451 ts_tablet_manager.cc:1398] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:10:51.624338 27455 tablet_bootstrap.cc:654] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.624702 27451 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:51.627059 27451 tablet_bootstrap.cc:654] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.628702 27454 tablet_bootstrap.cc:492] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:51.628998 27454 ts_tablet_manager.cc:1367] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240501 08:10:51.630066 27454 raft_consensus.cc:348] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.630504 27454 raft_consensus.cc:374] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.630674 27454 raft_consensus.cc:729] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.631109 27454 consensus_queue.cc:260] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.631841 27455 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:51.631948 27452 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:51.632102 27455 ts_tablet_manager.cc:1367] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240501 08:10:51.632143 27452 ts_tablet_manager.cc:1367] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20240501 08:10:51.633124 27452 raft_consensus.cc:348] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.633850 27452 raft_consensus.cc:374] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.633721 27454 ts_tablet_manager.cc:1398] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:10:51.633980 27452 raft_consensus.cc:729] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.633976 27455 raft_consensus.cc:348] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:51.634205 27455 raft_consensus.cc:374] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.634438 27455 raft_consensus.cc:729] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.634343 27452 consensus_queue.cc:260] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.634671 27454 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:51.635010 27452 ts_tablet_manager.cc:1398] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:10:51.634865 27455 consensus_queue.cc:260] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:51.636348 27455 ts_tablet_manager.cc:1398] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:10:51.636726 27454 tablet_bootstrap.cc:654] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.636761 27455 tablet_bootstrap.cc:492] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:51.636765 27452 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:51.638664 27452 tablet_bootstrap.cc:654] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.640209 27455 tablet_bootstrap.cc:654] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.644995 27451 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:51.645207 27454 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:51.645279 27451 ts_tablet_manager.cc:1367] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.003s
I20240501 08:10:51.645478 27454 ts_tablet_manager.cc:1367] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240501 08:10:51.646180 27451 raft_consensus.cc:348] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:51.646358 27454 raft_consensus.cc:348] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.646577 27454 raft_consensus.cc:374] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.646656 27454 raft_consensus.cc:729] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.646745 27451 raft_consensus.cc:374] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.646839 27451 raft_consensus.cc:729] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.647102 27454 consensus_queue.cc:260] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.647114 27451 consensus_queue.cc:260] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:51.647861 27454 ts_tablet_manager.cc:1398] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:51.647913 27451 ts_tablet_manager.cc:1398] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:51.648339 27451 tablet_bootstrap.cc:492] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:51.649706 27452 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:51.649963 27452 ts_tablet_manager.cc:1367] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240501 08:10:51.650842 27451 tablet_bootstrap.cc:654] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.650875 27452 raft_consensus.cc:348] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:51.651111 27452 raft_consensus.cc:374] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.651206 27452 raft_consensus.cc:729] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.651484 27452 consensus_queue.cc:260] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:51.652107 27452 ts_tablet_manager.cc:1398] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:10:51.653719 27455 tablet_bootstrap.cc:492] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:51.653920 27455 ts_tablet_manager.cc:1367] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 08:10:51.654891 27455 raft_consensus.cc:348] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.655398 27455 raft_consensus.cc:374] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.655444 27451 tablet_bootstrap.cc:492] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:51.655541 27455 raft_consensus.cc:729] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.655623 27451 ts_tablet_manager.cc:1367] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:10:51.655838 27455 consensus_queue.cc:260] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.656436 27451 raft_consensus.cc:348] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.656602 27451 raft_consensus.cc:374] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.656737 27451 raft_consensus.cc:729] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.656855 27455 ts_tablet_manager.cc:1398] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:51.656991 27451 consensus_queue.cc:260] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.658349 27451 ts_tablet_manager.cc:1398] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:10:51.710340 27354 raft_consensus.cc:482] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:51.710544 27354 raft_consensus.cc:504] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.711198 27354 leader_election.cc:288] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:10:51.711800 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c45caee3514df08b819cf61954f88d" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4" is_pre_election: true
I20240501 08:10:51.711844 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c45caee3514df08b819cf61954f88d" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943" is_pre_election: true
I20240501 08:10:51.712057 26602 raft_consensus.cc:2454] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 0.
I20240501 08:10:51.712046 26212 raft_consensus.cc:2454] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 0.
I20240501 08:10:51.712491 26408 leader_election.cc:302] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:51.712769 27354 raft_consensus.cc:2781] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:51.712872 27354 raft_consensus.cc:482] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:51.712955 27354 raft_consensus.cc:3037] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.714929 27397 raft_consensus.cc:482] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:51.715096 27397 raft_consensus.cc:504] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.716655 27397 leader_election.cc:288] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:10:51.724771 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981240ef4c1466cb37895e24a0c6e5c" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587" is_pre_election: true
I20240501 08:10:51.725091 26471 raft_consensus.cc:2454] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 0.
I20240501 08:10:51.725282 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981240ef4c1466cb37895e24a0c6e5c" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943" is_pre_election: true
I20240501 08:10:51.725535 26212 raft_consensus.cc:2454] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 0.
I20240501 08:10:51.725697 26278 leader_election.cc:302] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244; no voters:
I20240501 08:10:51.726810 27397 raft_consensus.cc:2781] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:51.726920 27397 raft_consensus.cc:482] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:51.727005 27397 raft_consensus.cc:3037] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.783083 27354 raft_consensus.cc:504] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.783124 27397 raft_consensus.cc:504] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.783847 27354 leader_election.cc:288] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 election: Requested vote from peers f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:10:51.783936 27397 leader_election.cc:288] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 1 election: Requested vote from peers 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:10:51.784473 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c45caee3514df08b819cf61954f88d" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
I20240501 08:10:51.784771 26602 raft_consensus.cc:3037] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.784736 26211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981240ef4c1466cb37895e24a0c6e5c" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943"
I20240501 08:10:51.784938 26211 raft_consensus.cc:3037] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.785118 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981240ef4c1466cb37895e24a0c6e5c" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
I20240501 08:10:51.785339 26471 raft_consensus.cc:3037] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.787860 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c45caee3514df08b819cf61954f88d" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943"
I20240501 08:10:51.788061 26212 raft_consensus.cc:3037] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.830724 27354 raft_consensus.cc:482] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:51.830936 27354 raft_consensus.cc:504] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:51.831597 27354 leader_election.cc:288] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:51.832260 26601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ddc36d3fc146938c366df2c7f090a6" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4" is_pre_election: true
I20240501 08:10:51.832355 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ddc36d3fc146938c366df2c7f090a6" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:10:51.832484 26601 raft_consensus.cc:2454] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 0.
I20240501 08:10:51.832580 26342 raft_consensus.cc:2454] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 0.
I20240501 08:10:51.832906 26405 leader_election.cc:302] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:51.833236 27354 raft_consensus.cc:2781] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:51.833371 27354 raft_consensus.cc:482] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:51.833463 27354 raft_consensus.cc:3037] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.886502 26471 raft_consensus.cc:2454] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 1.
I20240501 08:10:51.886502 26602 raft_consensus.cc:2454] T 99c45caee3514df08b819cf61954f88d P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
I20240501 08:10:51.886533 26211 raft_consensus.cc:2454] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 1.
I20240501 08:10:51.887516 26405 leader_election.cc:302] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:51.889580 26212 raft_consensus.cc:2454] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
I20240501 08:10:51.889539 27354 raft_consensus.cc:504] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:51.890028 26278 leader_election.cc:302] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:51.890380 27397 raft_consensus.cc:2781] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:51.890697 27469 raft_consensus.cc:2781] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:51.890897 27354 leader_election.cc:288] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 election: Requested vote from peers f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:51.891103 27397 raft_consensus.cc:686] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 1 LEADER]: Becoming Leader. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Running, Role: LEADER
I20240501 08:10:51.891559 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ddc36d3fc146938c366df2c7f090a6" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
I20240501 08:10:51.891798 26602 raft_consensus.cc:3037] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.891346 27469 raft_consensus.cc:686] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Becoming Leader. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Running, Role: LEADER
I20240501 08:10:51.891696 27397 consensus_queue.cc:237] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.892030 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ddc36d3fc146938c366df2c7f090a6" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:10:51.892241 26342 raft_consensus.cc:3037] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.892284 27469 consensus_queue.cc:237] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:51.897837 26070 catalog_manager.cc:5484] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: term changed from 0 to 1, leader changed from <none> to 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:51.897837 26067 catalog_manager.cc:5484] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc13665cb8894b1abf18738f527ab244 (127.25.107.2). New cstate: current_term: 1 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:51.981750 26602 raft_consensus.cc:2454] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
I20240501 08:10:51.982496 26342 raft_consensus.cc:2454] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
I20240501 08:10:51.982569 26405 leader_election.cc:302] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:51.983336 27469 raft_consensus.cc:2781] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:51.983503 27469 raft_consensus.cc:686] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Becoming Leader. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Running, Role: LEADER
I20240501 08:10:51.983783 27469 consensus_queue.cc:237] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:10:51.993633 26067 catalog_manager.cc:5484] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: term changed from 0 to 1, leader changed from <none> to 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:52.313735 27469 consensus_queue.cc:1035] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:52.319005 27469 consensus_queue.cc:1035] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:52.408764 27481 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:52.426719 27481 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:52.453704 27476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:52.454030 27476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:52.461592 27476 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 08:10:52.534356 27469 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:52.552525 27354 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 242.043 ms
time per row: 0.121021 ms
I20240501 08:10:52.846537 26070 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34800:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:10:52.860167 26452 tablet_service.cc:1467] Processing CreateTablet for tablet 359983c3db9a43be92e135c4b722dc0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.860563 26320 tablet_service.cc:1467] Processing CreateTablet for tablet 359983c3db9a43be92e135c4b722dc0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.860724 26452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359983c3db9a43be92e135c4b722dc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.861004 26320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359983c3db9a43be92e135c4b722dc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.861444 26190 tablet_service.cc:1467] Processing CreateTablet for tablet 359983c3db9a43be92e135c4b722dc0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.861894 26190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359983c3db9a43be92e135c4b722dc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.862371 26192 tablet_service.cc:1467] Processing CreateTablet for tablet c57c173fdd4444d6841410fc1adb7674 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.862788 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57c173fdd4444d6841410fc1adb7674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.862726 26450 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1a24c1d65242e9a634c7f1f99fff02 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.863133 26450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1a24c1d65242e9a634c7f1f99fff02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.866748 26191 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1a24c1d65242e9a634c7f1f99fff02 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.867254 26191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1a24c1d65242e9a634c7f1f99fff02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.870714 26322 tablet_service.cc:1467] Processing CreateTablet for tablet c57c173fdd4444d6841410fc1adb7674 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.871268 26322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57c173fdd4444d6841410fc1adb7674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.874876 26580 tablet_service.cc:1467] Processing CreateTablet for tablet c57c173fdd4444d6841410fc1adb7674 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.875433 26580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57c173fdd4444d6841410fc1adb7674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.878881 26582 tablet_service.cc:1467] Processing CreateTablet for tablet 0b1a24c1d65242e9a634c7f1f99fff02 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:52.879415 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b1a24c1d65242e9a634c7f1f99fff02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:52.880558 27542 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:52.882308 27542 tablet_bootstrap.cc:654] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.887477 27542 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:52.889456 27542 ts_tablet_manager.cc:1367] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240501 08:10:52.889274 27541 tablet_bootstrap.cc:492] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:52.890441 27542 raft_consensus.cc:348] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:52.890723 27542 raft_consensus.cc:374] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.890843 27542 raft_consensus.cc:729] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.891142 27542 consensus_queue.cc:260] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:52.893287 27542 ts_tablet_manager.cc:1398] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:10:52.893658 27542 tablet_bootstrap.cc:492] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:52.894809 27541 tablet_bootstrap.cc:654] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.895383 27542 tablet_bootstrap.cc:654] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.903950 27540 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:52.904700 27542 tablet_bootstrap.cc:492] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:52.904919 27542 ts_tablet_manager.cc:1367] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240501 08:10:52.905795 27540 tablet_bootstrap.cc:654] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.905798 27542 raft_consensus.cc:348] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:52.906064 27542 raft_consensus.cc:374] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.906122 27544 tablet_bootstrap.cc:492] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:52.906211 27542 raft_consensus.cc:729] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.906622 27542 consensus_queue.cc:260] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:52.907359 27542 ts_tablet_manager.cc:1398] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:10:52.907692 27542 tablet_bootstrap.cc:492] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:10:52.907691 27541 tablet_bootstrap.cc:492] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:52.907845 27541 ts_tablet_manager.cc:1367] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240501 08:10:52.909433 27542 tablet_bootstrap.cc:654] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.909402 27541 raft_consensus.cc:348] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:52.909633 27541 raft_consensus.cc:374] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.909739 27541 raft_consensus.cc:729] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.910056 27541 consensus_queue.cc:260] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:52.911062 27541 ts_tablet_manager.cc:1398] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:52.911469 27541 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: Bootstrap starting.
I20240501 08:10:52.914907 27544 tablet_bootstrap.cc:654] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.915575 27542 tablet_bootstrap.cc:492] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: No bootstrap required, opened a new log
I20240501 08:10:52.915740 27542 ts_tablet_manager.cc:1367] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 08:10:52.916445 27541 tablet_bootstrap.cc:654] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.916503 27542 raft_consensus.cc:348] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:52.916939 27542 raft_consensus.cc:374] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.917080 27544 tablet_bootstrap.cc:492] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:52.917116 27542 raft_consensus.cc:729] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.917466 27542 consensus_queue.cc:260] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:52.917644 27544 ts_tablet_manager.cc:1367] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 08:10:52.918552 27544 raft_consensus.cc:348] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:52.918845 27544 raft_consensus.cc:374] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.918969 27544 raft_consensus.cc:729] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.919153 27542 ts_tablet_manager.cc:1398] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:10:52.919420 27544 consensus_queue.cc:260] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:52.920197 27544 ts_tablet_manager.cc:1398] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:10:52.920539 27544 tablet_bootstrap.cc:492] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4: Bootstrap starting.
I20240501 08:10:52.922616 27544 tablet_bootstrap.cc:654] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.929244 27540 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:52.929441 27540 ts_tablet_manager.cc:1367] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.006s
I20240501 08:10:52.929723 27541 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: No bootstrap required, opened a new log
I20240501 08:10:52.929844 27544 tablet_bootstrap.cc:492] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4: No bootstrap required, opened a new log
I20240501 08:10:52.929888 27541 ts_tablet_manager.cc:1367] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240501 08:10:52.930023 27544 ts_tablet_manager.cc:1367] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240501 08:10:52.930356 27540 raft_consensus.cc:348] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:52.930579 27540 raft_consensus.cc:374] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.930570 27541 raft_consensus.cc:348] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:52.930696 27540 raft_consensus.cc:729] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.930796 27541 raft_consensus.cc:374] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.930899 27541 raft_consensus.cc:729] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.930806 27544 raft_consensus.cc:348] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:52.931010 27544 raft_consensus.cc:374] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.930938 27540 consensus_queue.cc:260] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:52.931090 27544 raft_consensus.cc:729] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.931180 27541 consensus_queue.cc:260] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:52.931380 27544 consensus_queue.cc:260] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:52.931933 27544 ts_tablet_manager.cc:1398] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:10:52.932556 27540 ts_tablet_manager.cc:1398] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:10:52.932571 27541 ts_tablet_manager.cc:1398] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:52.932893 27540 tablet_bootstrap.cc:492] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244: Bootstrap starting.
I20240501 08:10:52.934615 27540 tablet_bootstrap.cc:654] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:52.937695 27540 tablet_bootstrap.cc:492] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244: No bootstrap required, opened a new log
I20240501 08:10:52.937880 27540 ts_tablet_manager.cc:1367] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 08:10:52.938683 27540 raft_consensus.cc:348] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:52.938882 27540 raft_consensus.cc:374] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:52.938984 27540 raft_consensus.cc:729] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Initialized, Role: FOLLOWER
I20240501 08:10:52.939211 27540 consensus_queue.cc:260] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:52.939692 27540 ts_tablet_manager.cc:1398] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:10:52.943001 27341 raft_consensus.cc:482] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:52.943179 27341 raft_consensus.cc:504] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:52.943845 27341 leader_election.cc:288] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:52.944329 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:10:52.944330 26472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587" is_pre_election: true
I20240501 08:10:52.944581 26342 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 0.
I20240501 08:10:52.944581 26472 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 0.
I20240501 08:10:52.944994 26145 leader_election.cc:302] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:52.945269 27341 raft_consensus.cc:2781] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:52.945392 27341 raft_consensus.cc:482] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:52.945480 27341 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.947960 27341 raft_consensus.cc:504] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:52.948623 27341 leader_election.cc:288] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 election: Requested vote from peers 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:10:52.949064 26472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
I20240501 08:10:52.949167 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "c0624af4925d4686b5896f487f864943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:10:52.949311 26472 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.949391 26342 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:52.951537 26472 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 1.
I20240501 08:10:52.951552 26342 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0624af4925d4686b5896f487f864943 in term 1.
I20240501 08:10:52.952001 26148 leader_election.cc:302] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:52.952334 27341 raft_consensus.cc:2781] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:52.952704 27341 raft_consensus.cc:686] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 1 LEADER]: Becoming Leader. State: Replica: c0624af4925d4686b5896f487f864943, State: Running, Role: LEADER
I20240501 08:10:52.953003 27341 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:10:52.955984 26067 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0624af4925d4686b5896f487f864943 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "c0624af4925d4686b5896f487f864943" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:53.078948 27483 raft_consensus.cc:482] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:53.079169 27483 raft_consensus.cc:504] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:53.080024 27483 leader_election.cc:288] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:10:53.080543 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57c173fdd4444d6841410fc1adb7674" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943" is_pre_election: true
I20240501 08:10:53.080837 26212 raft_consensus.cc:2454] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 0.
I20240501 08:10:53.081452 26278 leader_election.cc:302] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:53.081777 27483 raft_consensus.cc:2781] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:53.081923 27483 raft_consensus.cc:482] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:53.082034 27483 raft_consensus.cc:3037] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:53.084726 27483 raft_consensus.cc:504] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:53.085366 27483 leader_election.cc:288] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 1 election: Requested vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:10:53.085955 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57c173fdd4444d6841410fc1adb7674" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943"
I20240501 08:10:53.086194 26212 raft_consensus.cc:3037] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:53.087544 26601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57c173fdd4444d6841410fc1adb7674" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4" is_pre_election: true
I20240501 08:10:53.087568 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57c173fdd4444d6841410fc1adb7674" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
I20240501 08:10:53.087822 26601 raft_consensus.cc:2454] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 0.
I20240501 08:10:53.088616 26212 raft_consensus.cc:2454] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 1.
I20240501 08:10:53.089042 26278 leader_election.cc:302] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, c0624af4925d4686b5896f487f864943; no voters: f0f978a65b1549689c460fce4699e7a4
I20240501 08:10:53.089437 27483 raft_consensus.cc:2781] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:53.089597 27483 raft_consensus.cc:686] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [term 1 LEADER]: Becoming Leader. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Running, Role: LEADER
I20240501 08:10:53.089879 27483 consensus_queue.cc:237] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:53.093000 27469 raft_consensus.cc:482] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:53.092940 26067 catalog_manager.cc:5484] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc13665cb8894b1abf18738f527ab244 (127.25.107.2). New cstate: current_term: 1 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:53.093194 27469 raft_consensus.cc:504] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:53.093909 27469 leader_election.cc:288] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:10:53.094455 26601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1a24c1d65242e9a634c7f1f99fff02" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4" is_pre_election: true
I20240501 08:10:53.094671 26601 raft_consensus.cc:2454] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 0.
I20240501 08:10:53.094786 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1a24c1d65242e9a634c7f1f99fff02" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943" is_pre_election: true
I20240501 08:10:53.095000 26212 raft_consensus.cc:2454] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 0.
I20240501 08:10:53.095149 26405 leader_election.cc:302] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, f0f978a65b1549689c460fce4699e7a4; no voters:
I20240501 08:10:53.095496 27469 raft_consensus.cc:2781] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:53.095592 27469 raft_consensus.cc:482] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:53.095700 27469 raft_consensus.cc:3037] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:53.097767 27469 raft_consensus.cc:504] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:53.098413 27469 leader_election.cc:288] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 election: Requested vote from peers f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:10:53.098903 26601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1a24c1d65242e9a634c7f1f99fff02" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
I20240501 08:10:53.098954 26212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b1a24c1d65242e9a634c7f1f99fff02" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0624af4925d4686b5896f487f864943"
I20240501 08:10:53.099136 26601 raft_consensus.cc:3037] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:53.099135 26212 raft_consensus.cc:3037] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:53.101990 26601 raft_consensus.cc:2454] T 0b1a24c1d65242e9a634c7f1f99fff02 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
I20240501 08:10:53.101991 26212 raft_consensus.cc:2454] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
I20240501 08:10:53.102675 26408 leader_election.cc:302] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, c0624af4925d4686b5896f487f864943; no voters:
I20240501 08:10:53.103034 27469 raft_consensus.cc:2781] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:53.103226 27469 raft_consensus.cc:686] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Becoming Leader. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Running, Role: LEADER
I20240501 08:10:53.103674 27469 consensus_queue.cc:237] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:10:53.106606 26067 catalog_manager.cc:5484] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: term changed from 0 to 1, leader changed from <none> to 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:53.351779 27553 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:53.363365 27565 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:53.503010 27559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:53.503369 27559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:53.511528 27559 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 08:10:53.605993 26601 raft_consensus.cc:3037] T c57c173fdd4444d6841410fc1adb7674 P f0f978a65b1549689c460fce4699e7a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:53.609735 27483 consensus_queue.cc:1035] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:53.622350 27354 consensus_queue.cc:1035] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:53.641809 27483 consensus_queue.cc:1035] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:53.677045 27354 consensus_queue.cc:1035] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:10:53.761420 26538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.1:40425, user_credentials={real_user=slave}} blocked reactor thread for 50870.1us
W20240501 08:10:53.764029 27601 log.cc:924] Time spent T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
INSERT report
rows total: 2000
time total: 281.18 ms
time per row: 0.14059 ms
I20240501 08:10:53.941892 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.5:43333
--local_ip_for_outbound_sockets=127.25.107.5
--tserver_master_addrs=127.25.107.62:37731
--webserver_port=46711
--webserver_interface=127.25.107.5
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 08:10:54.299603 27621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:54.299944 27621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:54.300086 27621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:54.308116 27621 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:54.308266 27621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:54.308429 27621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.5
I20240501 08:10:54.317849 27621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.5:43333
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.107.5
--webserver_port=46711
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.5
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:54.319293 27621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:54.321220 27621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:54.334453 27627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:54.334400 27629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:54.334693 27626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:54.334897 27621 server_base.cc:1034] running on GCE node
I20240501 08:10:54.336067 27621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:54.337119 27621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:54.338346 27621 hybrid_clock.cc:648] HybridClock initialized: now 1714551054338285 us; error 53 us; skew 500 ppm
I20240501 08:10:54.338688 27621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:54.346091 27621 webserver.cc:460] Webserver started at http://127.25.107.5:46711/ using document root <none> and password file <none>
I20240501 08:10:54.347041 27621 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:54.347195 27621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:54.353570 27621 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:10:54.356859 27635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.358450 27621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:54.358633 27621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0c0ceab7acf1424aa23a96d23e0aa786"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-dv41"
I20240501 08:10:54.359526 27621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:54.379237 27621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:54.380173 27621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:54.380484 27621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:54.381510 27621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:54.383430 27621 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:54.383528 27621 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.383690 27621 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:54.383769 27621 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.420665 27621 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.5:43333
I20240501 08:10:54.420748 27747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.5:43333 every 8 connection(s)
I20240501 08:10:54.422322 27621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:10:54.430871 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 27621
I20240501 08:10:54.431466 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.6:34341
--local_ip_for_outbound_sockets=127.25.107.6
--tserver_master_addrs=127.25.107.62:37731
--webserver_port=34541
--webserver_interface=127.25.107.6
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 08:10:54.437134 27748 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:54.437458 27748 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:54.438421 27748 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:54.440374 26067 ts_manager.cc:194] Re-registered known tserver with Master: 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5:43333)
I20240501 08:10:54.441349 26067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.5:52467
W20240501 08:10:54.778558 27752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:54.778865 27752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:54.778990 27752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:54.786643 27752 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:54.786767 27752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:54.786931 27752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.6
I20240501 08:10:54.796789 27752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.6:34341
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.107.6
--webserver_port=34541
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.6
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:54.798218 27752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:54.800164 27752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:54.812297 27759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:54.812387 27758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:54.812934 27761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:54.814134 27752 server_base.cc:1034] running on GCE node
I20240501 08:10:54.814818 27752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:54.816028 27752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:54.817636 27752 hybrid_clock.cc:648] HybridClock initialized: now 1714551054817500 us; error 137 us; skew 500 ppm
I20240501 08:10:54.818011 27752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:54.820494 27752 webserver.cc:460] Webserver started at http://127.25.107.6:34541/ using document root <none> and password file <none>
I20240501 08:10:54.821417 27752 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:54.821580 27752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:54.827401 27752 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 08:10:54.830868 27767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.832594 27752 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20240501 08:10:54.832785 27752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "f2a0e97dec014a17a4c96bfd2f60a96b"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-dv41"
I20240501 08:10:54.833760 27752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:54.849917 27752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:54.850843 27752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:54.851138 27752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:54.852195 27752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:54.853879 27752 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:54.853983 27752 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.854118 27752 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:54.854178 27752 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.890136 27752 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.6:34341
I20240501 08:10:54.890177 27879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.6:34341 every 8 connection(s)
I20240501 08:10:54.891594 27752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:10:54.892504 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 27752
I20240501 08:10:54.893265 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.7:36815
--local_ip_for_outbound_sockets=127.25.107.7
--tserver_master_addrs=127.25.107.62:37731
--webserver_port=45449
--webserver_interface=127.25.107.7
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 08:10:54.906368 27880 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:54.906675 27880 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:54.907568 27880 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:54.909106 26067 ts_manager.cc:194] Re-registered known tserver with Master: f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6:34341)
I20240501 08:10:54.910321 26067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.6:48267
W20240501 08:10:55.244696 27883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:55.245106 27883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:55.245268 27883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:55.253832 27883 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:55.254046 27883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:55.254191 27883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.7
I20240501 08:10:55.264304 27883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.7:36815
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.107.7
--webserver_port=45449
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.7
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:55.265729 27883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:55.267601 27883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:55.280187 27889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:55.280246 27888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:55.280201 27891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:55.281123 27883 server_base.cc:1034] running on GCE node
I20240501 08:10:55.281786 27883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:55.282938 27883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:55.284170 27883 hybrid_clock.cc:648] HybridClock initialized: now 1714551055284122 us; error 62 us; skew 500 ppm
I20240501 08:10:55.284539 27883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:55.287003 27883 webserver.cc:460] Webserver started at http://127.25.107.7:45449/ using document root <none> and password file <none>
I20240501 08:10:55.287801 27883 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:55.288026 27883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:55.294018 27883 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20240501 08:10:55.297147 27897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:55.298826 27883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:55.299015 27883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fba0b24a97a84f6fa331664028231d19"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-dv41"
I20240501 08:10:55.300061 27883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:55.322438 27883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:55.323460 27883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:55.323755 27883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:55.324723 27883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:55.326440 27883 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:55.326520 27883 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:55.326650 27883 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:55.326700 27883 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:55.359330 27883 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.7:36815
I20240501 08:10:55.359416 28009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.7:36815 every 8 connection(s)
I20240501 08:10:55.360865 27883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:10:55.363986 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 27883
I20240501 08:10:55.364598 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.8:33359
--local_ip_for_outbound_sockets=127.25.107.8
--tserver_master_addrs=127.25.107.62:37731
--webserver_port=45177
--webserver_interface=127.25.107.8
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 08:10:55.375237 28010 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:55.375584 28010 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:55.376505 28010 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:55.378212 26067 ts_manager.cc:194] Re-registered known tserver with Master: fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815)
I20240501 08:10:55.379436 26067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.7:47087
I20240501 08:10:55.444960 27748 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
W20240501 08:10:55.723237 28013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:55.723536 28013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:55.723641 28013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:55.731940 28013 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:55.732070 28013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:55.732203 28013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.8
I20240501 08:10:55.741776 28013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.8:33359
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.25.107.8
--webserver_port=45177
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.8
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:55.743211 28013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:55.747742 28013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:55.762498 28018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:55.762547 28021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:55.762461 28019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:55.767891 28013 server_base.cc:1034] running on GCE node
I20240501 08:10:55.768678 28013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:55.769869 28013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:55.771090 28013 hybrid_clock.cc:648] HybridClock initialized: now 1714551055771037 us; error 52 us; skew 500 ppm
I20240501 08:10:55.771502 28013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:55.774067 28013 webserver.cc:460] Webserver started at http://127.25.107.8:45177/ using document root <none> and password file <none>
I20240501 08:10:55.774911 28013 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:55.775041 28013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:55.781600 28013 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:10:55.784919 28027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:55.786540 28013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:55.786736 28013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "25e18e28f3914e45be7093bf5bf436e5"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-dv41"
I20240501 08:10:55.787658 28013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:55.820696 28013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:55.821663 28013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:55.821995 28013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:55.823133 28013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:55.824784 28013 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:55.824853 28013 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:55.824954 28013 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:55.825011 28013 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:55.859223 28013 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.8:33359
I20240501 08:10:55.859252 28139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.8:33359 every 8 connection(s)
I20240501 08:10:55.860873 28013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:10:55.864445 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 28013
I20240501 08:10:55.876446 28140 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:10:55.876850 28140 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:55.878064 28140 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:10:55.879827 26067 ts_manager.cc:194] Re-registered known tserver with Master: 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8:33359)
I20240501 08:10:55.880704 26067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.8:52075
I20240501 08:10:55.913292 27880 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
I20240501 08:10:56.382303 28010 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
I20240501 08:10:56.391993 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:56.418723 28075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:56.420240 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:10:56.451557 26322 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:10:56.480175 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:56.499275 27945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:56.519263 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:56.563829 26192 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:56.883812 28140 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
I20240501 08:10:57.084233 26063 ts_manager.cc:292] Set tserver state for c0624af4925d4686b5896f487f864943 to MAINTENANCE_MODE
W20240501 08:10:57.571875 28208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:57.572340 28208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:57.585489 28208 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:10:57.831218 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:57.848248 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:10:57.892035 28075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:57.905709 26322 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:10:57.935592 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:57.936096 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:57.965327 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:57.976527 28075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.011360 26192 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:58.025360 27945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.066077 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.099915 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:58.105347 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:10:58.106493 26322 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:10:58.114315 27945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.121778 26192 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:58.378815 28075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.385891 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:10:58.392306 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.415146 26322 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:10:58.427871 26192 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:58.464849 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:58.482941 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.525872 27945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.738902 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.758443 28075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.769966 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:10:58.788817 26322 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:10:58.805676 26192 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:58.828322 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:58.847939 27945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:58.851428 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:10:58.908946 28330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:58.909397 28330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:58.922005 28330 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:10:58.983634 26602 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:10:59.004510 26342 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:10:59.004633 26212 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:10:59.005904 27396 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:10:59.006913 27396 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:10:59.020448 27396 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } }
I20240501 08:10:59.026991 26341 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } }
I20240501 08:10:59.032152 26212 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } }
I20240501 08:10:59.034153 26062 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } }
I20240501 08:10:59.053499 26602 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:10:59.058892 26342 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:10:59.060431 26212 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:10:59.061276 28387 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:10:59.061734 27396 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:10:59.065037 28387 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:10:59.065580 26341 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:10:59.070456 26212 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:10:59.071969 26055 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12641e2d23ac450f9f04e41bb583b32d with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:10:59.072943 26067 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index 4 to 5, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:10:59.076880 26538 consensus_peers.cc:485] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 -> Peer fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815): Couldn't send request to peer fba0b24a97a84f6fa331664028231d19. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12641e2d23ac450f9f04e41bb583b32d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:10:59.147361 28075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:59.148052 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:59.159248 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:10:59.159278 26192 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:59.167028 26322 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:10:59.203295 27945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:59.214706 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:10:59.216130 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:59.466434 28433 ts_tablet_manager.cc:902] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: Initiating tablet copy from peer f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:10:59.468307 28433 tablet_copy_client.cc:308] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:37061
I20240501 08:10:59.494694 26622 tablet_copy_service.cc:140] P f0f978a65b1549689c460fce4699e7a4: Received BeginTabletCopySession request for tablet 12641e2d23ac450f9f04e41bb583b32d from peer fba0b24a97a84f6fa331664028231d19 ({username='slave'} at 127.25.107.7:43349)
I20240501 08:10:59.495009 26622 tablet_copy_service.cc:161] P f0f978a65b1549689c460fce4699e7a4: Beginning new tablet copy session on tablet 12641e2d23ac450f9f04e41bb583b32d from peer fba0b24a97a84f6fa331664028231d19 at {username='slave'} at 127.25.107.7:43349: session id = fba0b24a97a84f6fa331664028231d19-12641e2d23ac450f9f04e41bb583b32d
I20240501 08:10:59.499161 26622 tablet_copy_source_session.cc:205] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:10:59.504515 28433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12641e2d23ac450f9f04e41bb583b32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.516486 28433 tablet_copy_client.cc:783] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: tablet copy: Starting download of 0 data blocks...
I20240501 08:10:59.516929 28433 tablet_copy_client.cc:647] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: tablet copy: Starting download of 1 WAL segments...
I20240501 08:10:59.519784 28437 tablet_copy_client.cc:1002] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: tablet copy: Injecting latency into file download: 1500
W20240501 08:10:59.752244 28432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:59.752714 28432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:59.763576 28432 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:10:59.932557 28075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:59.944732 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:59.950709 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:00.000802 26322 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:00.004803 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:00.036216 26192 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:00.044975 27945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:00.059888 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240501 08:11:00.221917 28206 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 26130
I20240501 08:11:00.228772 26250 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:00.354969 28206 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 26130
W20240501 08:11:00.378217 26278 connection.cc:535] client connection to 127.25.107.1:40425 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:11:00.378535 26278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:11:00.380420 26278 consensus_peers.cc:485] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:00.384447 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:00.401306 26408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:00.402877 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:00.403057 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:00.467012 26538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:00.468765 26538 consensus_peers.cc:485] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:00.524004 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:00.643699 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:00.766391 26408 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:01.020418 28433 tablet_copy_client.cc:523] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:01.024178 28433 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: Bootstrap starting.
I20240501 08:11:01.034497 28433 log.cc:824] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:01.068782 28433 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:01.069324 28433 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: Bootstrap complete.
I20240501 08:11:01.069761 28433 ts_tablet_manager.cc:1367] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: Time spent bootstrapping tablet: real 0.046s user 0.040s sys 0.004s
I20240501 08:11:01.074860 28433 raft_consensus.cc:348] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:01.075335 28433 raft_consensus.cc:729] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fba0b24a97a84f6fa331664028231d19, State: Initialized, Role: LEARNER
I20240501 08:11:01.075907 28433 consensus_queue.cc:260] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:01.077415 28433 ts_tablet_manager.cc:1398] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20240501 08:11:01.078574 26622 tablet_copy_service.cc:342] P f0f978a65b1549689c460fce4699e7a4: Request end of tablet copy session fba0b24a97a84f6fa331664028231d19-12641e2d23ac450f9f04e41bb583b32d received from {username='slave'} at 127.25.107.7:43349
I20240501 08:11:01.078809 26622 tablet_copy_service.cc:433] P f0f978a65b1549689c460fce4699e7a4: ending tablet copy session fba0b24a97a84f6fa331664028231d19-12641e2d23ac450f9f04e41bb583b32d on tablet 12641e2d23ac450f9f04e41bb583b32d with peer fba0b24a97a84f6fa331664028231d19
I20240501 08:11:01.123832 27965 raft_consensus.cc:1203] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:11:01.365685 28291 raft_consensus.cc:482] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.365902 28291 raft_consensus.cc:504] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:11:01.366724 28291 leader_election.cc:288] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:11:01.367295 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
W20240501 08:11:01.368299 26408 leader_election.cc:334] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.368463 26408 leader_election.cc:302] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587; no voters: bc13665cb8894b1abf18738f527ab244, c0624af4925d4686b5896f487f864943
I20240501 08:11:01.368741 28291 raft_consensus.cc:2726] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:11:01.401357 28542 raft_consensus.cc:482] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.401559 28542 raft_consensus.cc:504] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:11:01.402287 28542 leader_election.cc:288] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:11:01.402765 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b85279a473435fb21f7e6c3cbebea2" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
W20240501 08:11:01.403684 26538 leader_election.cc:334] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.403841 26538 leader_election.cc:302] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0f978a65b1549689c460fce4699e7a4; no voters: bc13665cb8894b1abf18738f527ab244, c0624af4925d4686b5896f487f864943
I20240501 08:11:01.404107 28542 raft_consensus.cc:2726] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:11:01.426196 28291 raft_consensus.cc:482] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.426412 28291 raft_consensus.cc:504] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:01.427129 28291 leader_election.cc:288] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:11:01.427743 26600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc10e8d77e54d6f959552a962abee60" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4" is_pre_election: true
W20240501 08:11:01.428501 26408 leader_election.cc:334] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.428632 26408 leader_election.cc:302] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587; no voters: c0624af4925d4686b5896f487f864943, f0f978a65b1549689c460fce4699e7a4
I20240501 08:11:01.428850 28291 raft_consensus.cc:2726] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:11:01.472954 28542 raft_consensus.cc:482] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.473160 28542 raft_consensus.cc:504] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:01.473858 28542 leader_election.cc:288] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:11:01.474725 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc10e8d77e54d6f959552a962abee60" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587" is_pre_election: true
I20240501 08:11:01.474972 26471 raft_consensus.cc:2454] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 1.
W20240501 08:11:01.475078 26538 leader_election.cc:334] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.475368 26538 leader_election.cc:302] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, f0f978a65b1549689c460fce4699e7a4; no voters: c0624af4925d4686b5896f487f864943
I20240501 08:11:01.475610 28542 raft_consensus.cc:2781] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:11:01.475732 28542 raft_consensus.cc:482] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.475826 28542 raft_consensus.cc:3037] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:01.477701 28542 raft_consensus.cc:504] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:01.478340 28542 leader_election.cc:288] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 2 election: Requested vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:11:01.479074 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc10e8d77e54d6f959552a962abee60" candidate_uuid: "f0f978a65b1549689c460fce4699e7a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
I20240501 08:11:01.479321 26471 raft_consensus.cc:3037] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:11:01.479586 26538 leader_election.cc:334] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.481323 26471 raft_consensus.cc:2454] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0f978a65b1549689c460fce4699e7a4 in term 2.
I20240501 08:11:01.481760 26538 leader_election.cc:302] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, f0f978a65b1549689c460fce4699e7a4; no voters: c0624af4925d4686b5896f487f864943
I20240501 08:11:01.482045 28542 raft_consensus.cc:2781] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:01.482192 28542 raft_consensus.cc:686] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [term 2 LEADER]: Becoming Leader. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Running, Role: LEADER
I20240501 08:11:01.482518 28542 consensus_queue.cc:237] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:01.485330 26070 catalog_manager.cc:5484] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 reported cstate change: term changed from 1 to 2, leader changed from c0624af4925d4686b5896f487f864943 (127.25.107.1) to f0f978a65b1549689c460fce4699e7a4 (127.25.107.4). New cstate: current_term: 2 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:01.591301 28542 raft_consensus.cc:1050] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: attempting to promote NON_VOTER fba0b24a97a84f6fa331664028231d19 to VOTER
I20240501 08:11:01.592144 28542 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:01.593098 28527 raft_consensus.cc:482] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.593330 28527 raft_consensus.cc:504] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:11:01.594018 28527 leader_election.cc:288] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:11:01.594709 26341 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:01.594945 27965 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:01.595119 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587" is_pre_election: true
I20240501 08:11:01.595352 26471 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 1.
I20240501 08:11:01.595497 27396 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:01.595747 28501 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:01.595902 26278 leader_election.cc:302] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244; no voters:
W20240501 08:11:01.596387 26278 leader_election.cc:334] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.596432 28527 raft_consensus.cc:2781] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:11:01.596776 28527 raft_consensus.cc:482] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.596894 28527 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:11:01.598160 26538 consensus_peers.cc:485] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:01.599699 28527 raft_consensus.cc:504] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:11:01.600211 28527 leader_election.cc:288] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 election: Requested vote from peers 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:11:01.600246 27396 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:01.600979 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
I20240501 08:11:01.601226 26471 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:11:01.601533 26278 leader_election.cc:334] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.603992 26471 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 2.
I20240501 08:11:01.605127 26278 leader_election.cc:302] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244; no voters: c0624af4925d4686b5896f487f864943
I20240501 08:11:01.604965 26341 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:01.605012 27965 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:01.605581 26070 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index 5 to 6, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:01.606887 28527 raft_consensus.cc:2781] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:01.607828 28527 raft_consensus.cc:686] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Becoming Leader. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Running, Role: LEADER
I20240501 08:11:01.609428 28527 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:11:01.609715 28561 raft_consensus.cc:482] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.609902 28561 raft_consensus.cc:504] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:11:01.610874 28561 leader_election.cc:288] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:11:01.613512 26600 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:01.613721 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b85279a473435fb21f7e6c3cbebea2" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4" is_pre_election: true
I20240501 08:11:01.613937 26602 raft_consensus.cc:2454] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 1.
W20240501 08:11:01.613934 26278 leader_election.cc:334] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.615100 26275 leader_election.cc:302] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, f0f978a65b1549689c460fce4699e7a4; no voters: c0624af4925d4686b5896f487f864943
I20240501 08:11:01.615758 27965 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:11:01.615906 26341 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:11:01.615908 28527 raft_consensus.cc:2781] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:11:01.616304 28527 raft_consensus.cc:482] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.616434 28527 raft_consensus.cc:3037] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:01.616672 26062 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 reported cstate change: term changed from 1 to 2, leader changed from c0624af4925d4686b5896f487f864943 (127.25.107.1) to bc13665cb8894b1abf18738f527ab244 (127.25.107.2). New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:01.617750 28501 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:01.618160 27396 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:01.618832 28527 raft_consensus.cc:504] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:11:01.621129 27396 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:01.622247 28561 raft_consensus.cc:482] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.622421 28561 raft_consensus.cc:504] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:01.622961 28561 leader_election.cc:288] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:11:01.623064 28527 leader_election.cc:288] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 election: Requested vote from peers c0624af4925d4686b5896f487f864943 (127.25.107.1:40425), f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
W20240501 08:11:01.623214 26278 leader_election.cc:334] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.623322 27965 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:01.624120 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a56042bc14439b2ed2fa559e06a01" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "875f11971b814f2ea7ce89de1e4f4587" is_pre_election: true
I20240501 08:11:01.625115 26055 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12641e2d23ac450f9f04e41bb583b32d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:01.625242 26341 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
W20240501 08:11:01.626737 26278 leader_election.cc:334] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.626755 26600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b85279a473435fb21f7e6c3cbebea2" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
I20240501 08:11:01.626943 26600 raft_consensus.cc:3037] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:01.626940 26278 leader_election.cc:302] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc13665cb8894b1abf18738f527ab244; no voters: 875f11971b814f2ea7ce89de1e4f4587, c0624af4925d4686b5896f487f864943
I20240501 08:11:01.627408 26062 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:01.629765 26600 raft_consensus.cc:2454] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 2.
I20240501 08:11:01.630412 26275 leader_election.cc:302] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, f0f978a65b1549689c460fce4699e7a4; no voters: c0624af4925d4686b5896f487f864943
I20240501 08:11:01.630640 28527 raft_consensus.cc:2726] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:11:01.633677 26058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:01.635452 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 12641e2d23ac450f9f04e41bb583b32d: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.637327 28564 raft_consensus.cc:2781] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:01.637583 28564 raft_consensus.cc:686] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Becoming Leader. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Running, Role: LEADER
I20240501 08:11:01.638015 28564 consensus_queue.cc:237] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:11:01.642354 26062 catalog_manager.cc:5484] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 reported cstate change: term changed from 1 to 2, leader changed from c0624af4925d4686b5896f487f864943 (127.25.107.1) to bc13665cb8894b1abf18738f527ab244 (127.25.107.2). New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:01.678313 28291 raft_consensus.cc:482] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.678527 28291 raft_consensus.cc:504] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:01.679176 28291 leader_election.cc:288] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:11:01.679764 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a56042bc14439b2ed2fa559e06a01" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244" is_pre_election: true
I20240501 08:11:01.680040 26341 raft_consensus.cc:2454] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 1.
W20240501 08:11:01.680415 26408 leader_election.cc:334] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.680639 26405 leader_election.cc:302] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244; no voters: c0624af4925d4686b5896f487f864943
I20240501 08:11:01.680910 28291 raft_consensus.cc:2781] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:11:01.681031 28291 raft_consensus.cc:482] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0624af4925d4686b5896f487f864943)
I20240501 08:11:01.681116 28291 raft_consensus.cc:3037] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:01.683066 28291 raft_consensus.cc:504] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:01.683625 28291 leader_election.cc:288] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 election: Requested vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:11:01.684144 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a56042bc14439b2ed2fa559e06a01" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:11:01.684374 26341 raft_consensus.cc:3037] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:11:01.684943 26408 leader_election.cc:334] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:01.686406 26341 raft_consensus.cc:2454] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 2.
I20240501 08:11:01.686793 26405 leader_election.cc:302] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244; no voters: c0624af4925d4686b5896f487f864943
I20240501 08:11:01.687067 28291 raft_consensus.cc:2781] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:01.687206 28291 raft_consensus.cc:686] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [term 2 LEADER]: Becoming Leader. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Running, Role: LEADER
I20240501 08:11:01.687489 28291 consensus_queue.cc:237] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:01.690418 26062 catalog_manager.cc:5484] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: term changed from 1 to 2, leader changed from c0624af4925d4686b5896f487f864943 (127.25.107.1) to 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3). New cstate: current_term: 2 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:01.871490 26471 raft_consensus.cc:1261] T 5bc10e8d77e54d6f959552a962abee60 P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:11:01.872186 28500 consensus_queue.cc:1035] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:11:01.877605 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:01.960644 26341 consensus_queue.cc:237] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } }
I20240501 08:11:01.963703 26471 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:11:01.964202 26278 consensus_peers.cc:485] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:01.964532 28565 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:01.968020 28527 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } }
I20240501 08:11:01.968725 26471 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } }
I20240501 08:11:01.974488 26062 catalog_manager.cc:5484] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:01.995744 26341 consensus_queue.cc:237] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:02.000975 26471 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:02.005774 28527 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:02.009451 28527 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } } }
I20240501 08:11:02.013953 26471 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } } }
W20240501 08:11:02.014405 26278 consensus_peers.cc:485] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:02.018654 26055 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9981240ef4c1466cb37895e24a0c6e5c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:11:02.021235 26062 catalog_manager.cc:5484] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 4 to 5, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:02.030113 26278 consensus_peers.cc:485] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 -> Peer 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8:33359): Couldn't send request to peer 25e18e28f3914e45be7093bf5bf436e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9981240ef4c1466cb37895e24a0c6e5c. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:02.062888 26278 consensus_peers.cc:485] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:02.071552 26341 raft_consensus.cc:1261] T 697a56042bc14439b2ed2fa559e06a01 P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:11:02.072175 28510 consensus_queue.cc:1035] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:11:02.077108 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:02.077935 26600 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:11:02.078689 28527 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:11:02.122089 26278 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:02.197535 26471 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:11:02.198204 28527 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:11:02.486968 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:11:02.538031 28608 ts_tablet_manager.cc:902] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: Initiating tablet copy from peer bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:11:02.539773 28608 tablet_copy_client.cc:308] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:34319
I20240501 08:11:02.545997 26362 tablet_copy_service.cc:140] P bc13665cb8894b1abf18738f527ab244: Received BeginTabletCopySession request for tablet 9981240ef4c1466cb37895e24a0c6e5c from peer 25e18e28f3914e45be7093bf5bf436e5 ({username='slave'} at 127.25.107.8:42387)
I20240501 08:11:02.546232 26362 tablet_copy_service.cc:161] P bc13665cb8894b1abf18738f527ab244: Beginning new tablet copy session on tablet 9981240ef4c1466cb37895e24a0c6e5c from peer 25e18e28f3914e45be7093bf5bf436e5 at {username='slave'} at 127.25.107.8:42387: session id = 25e18e28f3914e45be7093bf5bf436e5-9981240ef4c1466cb37895e24a0c6e5c
I20240501 08:11:02.549074 26362 tablet_copy_source_session.cc:205] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:02.554033 28608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981240ef4c1466cb37895e24a0c6e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.565956 28608 tablet_copy_client.cc:783] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:02.566303 28608 tablet_copy_client.cc:647] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:02.568670 28612 tablet_copy_client.cc:1002] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Injecting latency into file download: 1500
W20240501 08:11:02.602370 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:02.849838 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 12641e2d23ac450f9f04e41bb583b32d: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:02.901312 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:02.947604 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:02.953862 26408 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:03.186445 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:03.918211 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 12641e2d23ac450f9f04e41bb583b32d: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:04.069203 28608 tablet_copy_client.cc:523] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:04.072588 28608 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap starting.
I20240501 08:11:04.082199 28608 log.cc:824] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:04.115217 28608 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:04.115682 28608 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap complete.
I20240501 08:11:04.116086 28608 ts_tablet_manager.cc:1367] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: Time spent bootstrapping tablet: real 0.044s user 0.039s sys 0.004s
I20240501 08:11:04.120609 28608 raft_consensus.cc:348] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:04.121034 28608 raft_consensus.cc:729] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 25e18e28f3914e45be7093bf5bf436e5, State: Initialized, Role: LEARNER
I20240501 08:11:04.121533 28608 consensus_queue.cc:260] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:04.123294 28608 ts_tablet_manager.cc:1398] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 08:11:04.124327 26362 tablet_copy_service.cc:342] P bc13665cb8894b1abf18738f527ab244: Request end of tablet copy session 25e18e28f3914e45be7093bf5bf436e5-9981240ef4c1466cb37895e24a0c6e5c received from {username='slave'} at 127.25.107.8:42387
I20240501 08:11:04.124480 26362 tablet_copy_service.cc:433] P bc13665cb8894b1abf18738f527ab244: ending tablet copy session 25e18e28f3914e45be7093bf5bf436e5-9981240ef4c1466cb37895e24a0c6e5c on tablet 9981240ef4c1466cb37895e24a0c6e5c with peer 25e18e28f3914e45be7093bf5bf436e5
W20240501 08:11:04.386426 26278 consensus_peers.cc:485] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:11:04.536401 28095 raft_consensus.cc:1203] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 08:11:04.569272 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:04.616937 26278 consensus_peers.cc:485] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:04.617652 26278 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:04.663417 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:04.888789 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:11:04.932310 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:11:04.935511 28679 raft_consensus.cc:1050] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244: attempting to promote NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 to VOTER
I20240501 08:11:04.936272 28679 consensus_queue.cc:237] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } }
I20240501 08:11:04.939234 26471 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:04.939508 28095 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 LEARNER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:11:04.939774 26278 consensus_peers.cc:485] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:04.940197 28680 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:04.940461 28679 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:04.944427 28679 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:04.945308 28095 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:04.948239 26471 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:04.949335 26070 catalog_manager.cc:5484] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 5 to 6, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:04.956020 26341 consensus_queue.cc:237] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } }
I20240501 08:11:04.957984 28095 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:11:04.958199 26471 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:11:04.959116 28679 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:04.959440 28643 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:04.962174 28680 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:04.966159 28095 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:04.966516 26055 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9981240ef4c1466cb37895e24a0c6e5c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:04.966063 26471 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:04.967521 26069 catalog_manager.cc:5484] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:11:04.978113 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9981240ef4c1466cb37895e24a0c6e5c: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:05.206998 26341 consensus_queue.cc:237] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:11:05.209259 26600 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:05.209839 28682 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:11:05.210085 26278 consensus_peers.cc:485] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:05.212353 28680 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } }
I20240501 08:11:05.212937 26600 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } }
I20240501 08:11:05.217037 26070 catalog_manager.cc:5484] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:05.221704 26341 consensus_queue.cc:237] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:05.224587 26600 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:05.225994 28682 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:11:05.226238 26278 consensus_peers.cc:485] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 -> Peer 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8:33359): Couldn't send request to peer 25e18e28f3914e45be7093bf5bf436e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5b85279a473435fb21f7e6c3cbebea2. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:05.226457 26278 consensus_peers.cc:485] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:05.228381 28682 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } } }
I20240501 08:11:05.229112 26600 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } } }
I20240501 08:11:05.232156 26055 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a5b85279a473435fb21f7e6c3cbebea2 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:11:05.233750 26070 catalog_manager.cc:5484] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 5 to 6, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:05.274880 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:11:05.275777 26408 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:11:05.429617 26408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 08:11:05.490057 26538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 08:11:05.491021 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:11:05.637001 28714 ts_tablet_manager.cc:902] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: Initiating tablet copy from peer bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:11:05.638554 28714 tablet_copy_client.cc:308] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:34319
I20240501 08:11:05.639333 26362 tablet_copy_service.cc:140] P bc13665cb8894b1abf18738f527ab244: Received BeginTabletCopySession request for tablet a5b85279a473435fb21f7e6c3cbebea2 from peer 25e18e28f3914e45be7093bf5bf436e5 ({username='slave'} at 127.25.107.8:42387)
I20240501 08:11:05.639511 26362 tablet_copy_service.cc:161] P bc13665cb8894b1abf18738f527ab244: Beginning new tablet copy session on tablet a5b85279a473435fb21f7e6c3cbebea2 from peer 25e18e28f3914e45be7093bf5bf436e5 at {username='slave'} at 127.25.107.8:42387: session id = 25e18e28f3914e45be7093bf5bf436e5-a5b85279a473435fb21f7e6c3cbebea2
I20240501 08:11:05.641707 26362 tablet_copy_source_session.cc:205] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:05.643026 28714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b85279a473435fb21f7e6c3cbebea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:05.648422 28714 tablet_copy_client.cc:783] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:05.648741 28714 tablet_copy_client.cc:647] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Starting download of 1 WAL segments...
W20240501 08:11:05.650465 26278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 44 similar messages]
I20240501 08:11:05.651343 28718 tablet_copy_client.cc:1002] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Injecting latency into file download: 1500
W20240501 08:11:05.804687 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:11:06.017979 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 12641e2d23ac450f9f04e41bb583b32d: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:06.910492 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:11:07.041544 26278 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:11:07.151796 28714 tablet_copy_client.cc:523] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:07.154826 28714 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap starting.
I20240501 08:11:07.187585 28714 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:07.188023 28714 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap complete.
W20240501 08:11:07.188056 26058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 15 similar messages]
I20240501 08:11:07.188333 28714 ts_tablet_manager.cc:1367] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: Time spent bootstrapping tablet: real 0.034s user 0.028s sys 0.004s
W20240501 08:11:07.189292 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9981240ef4c1466cb37895e24a0c6e5c: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:07.189361 28714 raft_consensus.cc:348] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:07.189626 28714 raft_consensus.cc:729] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 25e18e28f3914e45be7093bf5bf436e5, State: Initialized, Role: LEARNER
I20240501 08:11:07.189790 28714 consensus_queue.cc:260] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:07.190469 28714 ts_tablet_manager.cc:1398] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:11:07.191318 26362 tablet_copy_service.cc:342] P bc13665cb8894b1abf18738f527ab244: Request end of tablet copy session 25e18e28f3914e45be7093bf5bf436e5-a5b85279a473435fb21f7e6c3cbebea2 received from {username='slave'} at 127.25.107.8:42387
I20240501 08:11:07.191457 26362 tablet_copy_service.cc:433] P bc13665cb8894b1abf18738f527ab244: ending tablet copy session 25e18e28f3914e45be7093bf5bf436e5-a5b85279a473435fb21f7e6c3cbebea2 on tablet a5b85279a473435fb21f7e6c3cbebea2 with peer 25e18e28f3914e45be7093bf5bf436e5
W20240501 08:11:07.219274 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:11:07.454622 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:11:07.598085 28095 raft_consensus.cc:1203] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240501 08:11:07.613814 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:11:07.697180 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:11:07.831152 26278 consensus_peers.cc:485] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:11:08.069628 26408 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:11:08.203127 28792 raft_consensus.cc:1050] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244: attempting to promote NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 to VOTER
I20240501 08:11:08.203841 28792 consensus_queue.cc:237] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } }
I20240501 08:11:08.206465 26600 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:11:08.207221 28791 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:08.207392 28095 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 LEARNER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240501 08:11:08.207523 26278 consensus_peers.cc:485] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:08.207922 28791 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:08.210345 28794 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:08.211057 28095 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
W20240501 08:11:08.211977 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:11:08.214753 26600 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:08.216106 26070 catalog_manager.cc:5484] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 6 to 7, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:08.222895 26341 consensus_queue.cc:237] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } }
I20240501 08:11:08.224761 28095 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:11:08.225122 26600 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:11:08.225451 28791 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:08.225706 28794 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:08.228592 28792 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:08.229246 28095 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:08.229362 26600 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:08.233422 26069 catalog_manager.cc:5484] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 7 to 8, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:08.233973 26055 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a5b85279a473435fb21f7e6c3cbebea2 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 08:11:08.239800 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a5b85279a473435fb21f7e6c3cbebea2: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:08.294772 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:11:08.591995 26341 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:11:08.594293 26472 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240501 08:11:08.594904 26278 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:08.594949 28791 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:08.597328 28792 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } }
I20240501 08:11:08.597944 26472 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } }
I20240501 08:11:08.602342 26070 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:08.606794 26341 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
W20240501 08:11:08.610117 26278 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 -> Peer 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8:33359): Couldn't send request to peer 25e18e28f3914e45be7093bf5bf436e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 359983c3db9a43be92e135c4b722dc0f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:08.610769 26472 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240501 08:11:08.611541 26278 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:08.611850 28792 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:08.614538 28791 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } } }
I20240501 08:11:08.615242 26472 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } } }
I20240501 08:11:08.618652 26055 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 359983c3db9a43be92e135c4b722dc0f with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:11:08.620811 26069 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 5 to 6, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } } }
I20240501 08:11:08.991972 28827 ts_tablet_manager.cc:902] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: Initiating tablet copy from peer bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:11:08.994053 28827 tablet_copy_client.cc:308] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:34319
I20240501 08:11:08.994796 26362 tablet_copy_service.cc:140] P bc13665cb8894b1abf18738f527ab244: Received BeginTabletCopySession request for tablet 359983c3db9a43be92e135c4b722dc0f from peer 25e18e28f3914e45be7093bf5bf436e5 ({username='slave'} at 127.25.107.8:42387)
I20240501 08:11:08.994987 26362 tablet_copy_service.cc:161] P bc13665cb8894b1abf18738f527ab244: Beginning new tablet copy session on tablet 359983c3db9a43be92e135c4b722dc0f from peer 25e18e28f3914e45be7093bf5bf436e5 at {username='slave'} at 127.25.107.8:42387: session id = 25e18e28f3914e45be7093bf5bf436e5-359983c3db9a43be92e135c4b722dc0f
I20240501 08:11:08.997956 26362 tablet_copy_source_session.cc:205] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:08.999246 28827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359983c3db9a43be92e135c4b722dc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:09.003665 28827 tablet_copy_client.cc:783] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:09.003885 28827 tablet_copy_client.cc:647] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:09.005419 28831 tablet_copy_client.cc:1002] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Injecting latency into file download: 1500
W20240501 08:11:09.265415 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9981240ef4c1466cb37895e24a0c6e5c: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:09.315380 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:11:09.563612 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:11:09.856952 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:11:09.928817 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:11:10.204771 26408 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:11:10.451298 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a5b85279a473435fb21f7e6c3cbebea2: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:10.474303 26408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 08:11:10.505966 28827 tablet_copy_client.cc:523] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:10.508591 28827 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap starting.
W20240501 08:11:10.529106 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:11:10.541373 28827 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:10.541738 28827 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap complete.
I20240501 08:11:10.541965 28827 ts_tablet_manager.cc:1367] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: Time spent bootstrapping tablet: real 0.033s user 0.030s sys 0.001s
I20240501 08:11:10.542670 28827 raft_consensus.cc:348] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:10.542871 28827 raft_consensus.cc:729] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 25e18e28f3914e45be7093bf5bf436e5, State: Initialized, Role: LEARNER
I20240501 08:11:10.543033 28827 consensus_queue.cc:260] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:10.543754 28827 ts_tablet_manager.cc:1398] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 08:11:10.544571 26362 tablet_copy_service.cc:342] P bc13665cb8894b1abf18738f527ab244: Request end of tablet copy session 25e18e28f3914e45be7093bf5bf436e5-359983c3db9a43be92e135c4b722dc0f received from {username='slave'} at 127.25.107.8:42387
I20240501 08:11:10.544720 26362 tablet_copy_service.cc:433] P bc13665cb8894b1abf18738f527ab244: ending tablet copy session 25e18e28f3914e45be7093bf5bf436e5-359983c3db9a43be92e135c4b722dc0f on tablet 359983c3db9a43be92e135c4b722dc0f with peer 25e18e28f3914e45be7093bf5bf436e5
W20240501 08:11:10.570361 26538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:11:10.571251 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:11:10.819977 26278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240501 08:11:10.988965 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:11:11.011911 28095 raft_consensus.cc:1203] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240501 08:11:11.200035 26278 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:11:11.406287 28900 raft_consensus.cc:1050] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: attempting to promote NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 to VOTER
I20240501 08:11:11.407202 28900 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } }
I20240501 08:11:11.410236 28095 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 LEARNER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:11:11.410940 28900 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:11.411154 26472 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240501 08:11:11.411746 26278 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:11.411775 28900 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:11.419226 28901 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:11.419900 26472 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:11.423298 28095 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:11.423969 26070 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 6 to 7, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:11.430473 26341 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } }
I20240501 08:11:11.432183 28095 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:11:11.432269 26472 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:11:11.433405 28849 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:11.433657 28900 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:11.435964 28901 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:11.436502 26472 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:11.439394 26055 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 359983c3db9a43be92e135c4b722dc0f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:11.439734 28095 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:11.440804 26062 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 7 to 8, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:11:11.480839 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 359983c3db9a43be92e135c4b722dc0f: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:11.771625 26472 consensus_queue.cc:237] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:11.774015 26341 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:11:11.774384 26408 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:11.774570 28678 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:11.777407 28924 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } }
I20240501 08:11:11.777844 26341 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } }
I20240501 08:11:11.782107 26070 catalog_manager.cc:5484] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:11.786585 26472 consensus_queue.cc:237] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:11.788779 26341 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:11.789394 28930 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:11:11.790902 26408 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:11.791762 28930 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:11.792397 26341 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:11.795903 26058 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 595048c2a092461b979fe7e107882d75 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 08:11:11.797793 26069 catalog_manager.cc:5484] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 4 to 5, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:11.799258 26408 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815): Couldn't send request to peer fba0b24a97a84f6fa331664028231d19. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 595048c2a092461b979fe7e107882d75. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:11.918534 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:11:12.087153 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:11:12.350814 28947 ts_tablet_manager.cc:902] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: Initiating tablet copy from peer 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:11:12.352021 28947 tablet_copy_client.cc:308] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:41015
I20240501 08:11:12.357754 26492 tablet_copy_service.cc:140] P 875f11971b814f2ea7ce89de1e4f4587: Received BeginTabletCopySession request for tablet 595048c2a092461b979fe7e107882d75 from peer fba0b24a97a84f6fa331664028231d19 ({username='slave'} at 127.25.107.7:47561)
I20240501 08:11:12.358022 26492 tablet_copy_service.cc:161] P 875f11971b814f2ea7ce89de1e4f4587: Beginning new tablet copy session on tablet 595048c2a092461b979fe7e107882d75 from peer fba0b24a97a84f6fa331664028231d19 at {username='slave'} at 127.25.107.7:47561: session id = fba0b24a97a84f6fa331664028231d19-595048c2a092461b979fe7e107882d75
I20240501 08:11:12.360584 26492 tablet_copy_source_session.cc:205] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:12.362138 28947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595048c2a092461b979fe7e107882d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:12.366834 28947 tablet_copy_client.cc:783] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:12.367094 28947 tablet_copy_client.cc:647] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:12.368851 28951 tablet_copy_client.cc:1002] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: tablet copy: Injecting latency into file download: 1500
W20240501 08:11:12.524252 26058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240501 08:11:12.525089 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a5b85279a473435fb21f7e6c3cbebea2: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:12.581125 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:11:12.688872 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:11:12.863629 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:11:13.162956 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:11:13.432214 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:11:13.621479 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 359983c3db9a43be92e135c4b722dc0f: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:13.869189 28947 tablet_copy_client.cc:523] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.871750 28947 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: Bootstrap starting.
I20240501 08:11:13.902885 28947 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.903355 28947 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: Bootstrap complete.
I20240501 08:11:13.903630 28947 ts_tablet_manager.cc:1367] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.004s
I20240501 08:11:13.904569 28947 raft_consensus.cc:348] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:13.904788 28947 raft_consensus.cc:729] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fba0b24a97a84f6fa331664028231d19, State: Initialized, Role: LEARNER
I20240501 08:11:13.905016 28947 consensus_queue.cc:260] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:13.905651 28947 ts_tablet_manager.cc:1398] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:13.906407 26492 tablet_copy_service.cc:342] P 875f11971b814f2ea7ce89de1e4f4587: Request end of tablet copy session fba0b24a97a84f6fa331664028231d19-595048c2a092461b979fe7e107882d75 received from {username='slave'} at 127.25.107.7:47561
I20240501 08:11:13.906560 26492 tablet_copy_service.cc:433] P 875f11971b814f2ea7ce89de1e4f4587: ending tablet copy session fba0b24a97a84f6fa331664028231d19-595048c2a092461b979fe7e107882d75 on tablet 595048c2a092461b979fe7e107882d75 with peer fba0b24a97a84f6fa331664028231d19
W20240501 08:11:14.113759 26408 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:11:14.252554 27965 raft_consensus.cc:1203] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 08:11:14.435120 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:11:14.673497 28983 raft_consensus.cc:1050] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: attempting to promote NON_VOTER fba0b24a97a84f6fa331664028231d19 to VOTER
I20240501 08:11:14.674492 28983 consensus_queue.cc:237] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:14.677074 26341 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:14.677130 27965 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:14.677668 29022 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:14.678167 29025 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:11:14.678570 26408 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:14.683276 29022 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:14.683908 26341 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:14.687414 27965 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:14.688809 26070 catalog_manager.cc:5484] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 5 to 6, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.694542 26471 consensus_queue.cc:237] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:14.696051 26341 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:11:14.696612 27965 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:11:14.696650 29023 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:14.697121 29025 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:14.700136 29022 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:14.700065 26342 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:14.700188 27965 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:14.703438 26058 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 595048c2a092461b979fe7e107882d75 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:14.704350 26062 catalog_manager.cc:5484] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 6 to 7, VOTER c0624af4925d4686b5896f487f864943 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:11:14.710726 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 595048c2a092461b979fe7e107882d75: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:14.881155 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:11:14.884881 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:11:14.985764 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:11:15.042467 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.049005 26450 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:15.050038 28075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.098903 26582 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:15.100507 26322 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:15.106578 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.120458 27945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:11:15.173386 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:11:15.279527 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.291100 26322 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:15.296061 28075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.296308 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:15.299304 26582 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:15.310765 26450 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:15.325378 27945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:11:15.401895 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:11:15.410336 26342 consensus_queue.cc:237] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:15.412547 28094 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:15.414141 28849 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:11:15.423216 26276 consensus_peers.cc:485] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 -> Peer 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5:43333): Couldn't send request to peer 0c0ceab7acf1424aa23a96d23e0aa786. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9981240ef4c1466cb37895e24a0c6e5c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:15.424717 26471 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:15.425215 29121 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:15.431428 28849 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:15.436161 28095 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:15.442989 26062 catalog_manager.cc:5484] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 7 to 8, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:15.450127 26471 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:15.460722 26600 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:15.463589 27965 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:15.464216 28500 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:15.468948 26342 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:15.468587 28500 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:15.469805 27965 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
W20240501 08:11:15.472268 26536 consensus_peers.cc:485] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 -> Peer f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6:34341): Couldn't send request to peer f2a0e97dec014a17a4c96bfd2f60a96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12641e2d23ac450f9f04e41bb583b32d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:15.473330 29138 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:15.474095 26067 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index 7 to 8, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:15.475617 26342 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:15.493058 26342 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:15.495476 26471 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:11:15.496167 28095 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:11:15.496104 28849 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:15.496627 28849 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:15.498769 29121 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:15.499927 26471 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
W20240501 08:11:15.502420 26278 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 -> Peer fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815): Couldn't send request to peer fba0b24a97a84f6fa331664028231d19. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 359983c3db9a43be92e135c4b722dc0f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:15.505384 26062 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 8 to 9, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:15.512084 28095 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
W20240501 08:11:15.726013 26408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:11:15.757491 26538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:11:15.842736 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:11:15.876482 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 595048c2a092461b979fe7e107882d75: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:15.883915 29180 ts_tablet_manager.cc:902] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: Initiating tablet copy from peer bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:11:15.885560 29180 tablet_copy_client.cc:308] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:34319
I20240501 08:11:15.891330 26362 tablet_copy_service.cc:140] P bc13665cb8894b1abf18738f527ab244: Received BeginTabletCopySession request for tablet 9981240ef4c1466cb37895e24a0c6e5c from peer 0c0ceab7acf1424aa23a96d23e0aa786 ({username='slave'} at 127.25.107.5:40265)
I20240501 08:11:15.891531 26362 tablet_copy_service.cc:161] P bc13665cb8894b1abf18738f527ab244: Beginning new tablet copy session on tablet 9981240ef4c1466cb37895e24a0c6e5c from peer 0c0ceab7acf1424aa23a96d23e0aa786 at {username='slave'} at 127.25.107.5:40265: session id = 0c0ceab7acf1424aa23a96d23e0aa786-9981240ef4c1466cb37895e24a0c6e5c
I20240501 08:11:15.893939 26362 tablet_copy_source_session.cc:205] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:15.897886 29180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981240ef4c1466cb37895e24a0c6e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:15.907862 29180 tablet_copy_client.cc:783] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:15.908178 29180 tablet_copy_client.cc:647] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:15.909997 29184 tablet_copy_client.cc:1002] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Injecting latency into file download: 1500
I20240501 08:11:15.918464 29186 ts_tablet_manager.cc:902] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: Initiating tablet copy from peer f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:11:15.919960 29186 tablet_copy_client.cc:308] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:37061
I20240501 08:11:15.925850 26622 tablet_copy_service.cc:140] P f0f978a65b1549689c460fce4699e7a4: Received BeginTabletCopySession request for tablet 12641e2d23ac450f9f04e41bb583b32d from peer f2a0e97dec014a17a4c96bfd2f60a96b ({username='slave'} at 127.25.107.6:48597)
I20240501 08:11:15.926016 26622 tablet_copy_service.cc:161] P f0f978a65b1549689c460fce4699e7a4: Beginning new tablet copy session on tablet 12641e2d23ac450f9f04e41bb583b32d from peer f2a0e97dec014a17a4c96bfd2f60a96b at {username='slave'} at 127.25.107.6:48597: session id = f2a0e97dec014a17a4c96bfd2f60a96b-12641e2d23ac450f9f04e41bb583b32d
I20240501 08:11:15.928141 26622 tablet_copy_source_session.cc:205] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:15.931941 29186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12641e2d23ac450f9f04e41bb583b32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:15.941834 29186 tablet_copy_client.cc:783] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:15.942103 29186 tablet_copy_client.cc:647] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:15.943881 29190 tablet_copy_client.cc:1002] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Injecting latency into file download: 1500
W20240501 08:11:16.083581 26278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240501 08:11:16.105417 29211 ts_tablet_manager.cc:902] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: Initiating tablet copy from peer bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:11:16.106762 29211 tablet_copy_client.cc:308] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:34319
I20240501 08:11:16.112264 26362 tablet_copy_service.cc:140] P bc13665cb8894b1abf18738f527ab244: Received BeginTabletCopySession request for tablet 359983c3db9a43be92e135c4b722dc0f from peer fba0b24a97a84f6fa331664028231d19 ({username='slave'} at 127.25.107.7:54191)
I20240501 08:11:16.112411 26362 tablet_copy_service.cc:161] P bc13665cb8894b1abf18738f527ab244: Beginning new tablet copy session on tablet 359983c3db9a43be92e135c4b722dc0f from peer fba0b24a97a84f6fa331664028231d19 at {username='slave'} at 127.25.107.7:54191: session id = fba0b24a97a84f6fa331664028231d19-359983c3db9a43be92e135c4b722dc0f
I20240501 08:11:16.114367 26362 tablet_copy_source_session.cc:205] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:16.115456 29211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359983c3db9a43be92e135c4b722dc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:16.119736 29211 tablet_copy_client.cc:783] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:16.119943 29211 tablet_copy_client.cc:647] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:16.121456 29215 tablet_copy_client.cc:1002] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: tablet copy: Injecting latency into file download: 1500
W20240501 08:11:16.885187 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:11:16.939450 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 595048c2a092461b979fe7e107882d75: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:17.341305 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:11:17.410686 29180 tablet_copy_client.cc:523] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:17.414057 29180 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap starting.
I20240501 08:11:17.424228 29180 log.cc:824] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:17.446818 29186 tablet_copy_client.cc:523] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:17.450132 29186 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap starting.
I20240501 08:11:17.457792 29180 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:17.458281 29180 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap complete.
I20240501 08:11:17.458678 29180 ts_tablet_manager.cc:1367] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: Time spent bootstrapping tablet: real 0.045s user 0.042s sys 0.005s
I20240501 08:11:17.460310 29186 log.cc:824] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:17.462990 29180 raft_consensus.cc:348] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:17.463346 29180 raft_consensus.cc:729] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0c0ceab7acf1424aa23a96d23e0aa786, State: Initialized, Role: LEARNER
I20240501 08:11:17.463778 29180 consensus_queue.cc:260] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:17.465274 29180 ts_tablet_manager.cc:1398] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 08:11:17.466221 26362 tablet_copy_service.cc:342] P bc13665cb8894b1abf18738f527ab244: Request end of tablet copy session 0c0ceab7acf1424aa23a96d23e0aa786-9981240ef4c1466cb37895e24a0c6e5c received from {username='slave'} at 127.25.107.5:40265
I20240501 08:11:17.466414 26362 tablet_copy_service.cc:433] P bc13665cb8894b1abf18738f527ab244: ending tablet copy session 0c0ceab7acf1424aa23a96d23e0aa786-9981240ef4c1466cb37895e24a0c6e5c on tablet 9981240ef4c1466cb37895e24a0c6e5c with peer 0c0ceab7acf1424aa23a96d23e0aa786
I20240501 08:11:17.495551 29186 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:17.496038 29186 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap complete.
I20240501 08:11:17.496491 29186 ts_tablet_manager.cc:1367] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: Time spent bootstrapping tablet: real 0.047s user 0.047s sys 0.000s
I20240501 08:11:17.500921 29186 raft_consensus.cc:348] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:17.501303 29186 raft_consensus.cc:729] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f2a0e97dec014a17a4c96bfd2f60a96b, State: Initialized, Role: LEARNER
I20240501 08:11:17.501778 29186 consensus_queue.cc:260] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:17.503319 29186 ts_tablet_manager.cc:1398] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240501 08:11:17.504242 26622 tablet_copy_service.cc:342] P f0f978a65b1549689c460fce4699e7a4: Request end of tablet copy session f2a0e97dec014a17a4c96bfd2f60a96b-12641e2d23ac450f9f04e41bb583b32d received from {username='slave'} at 127.25.107.6:48597
I20240501 08:11:17.504369 26622 tablet_copy_service.cc:433] P f0f978a65b1549689c460fce4699e7a4: ending tablet copy session f2a0e97dec014a17a4c96bfd2f60a96b-12641e2d23ac450f9f04e41bb583b32d on tablet 12641e2d23ac450f9f04e41bb583b32d with peer f2a0e97dec014a17a4c96bfd2f60a96b
I20240501 08:11:17.600298 27703 raft_consensus.cc:1203] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240501 08:11:17.613149 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:11:17.621853 29211 tablet_copy_client.cc:523] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:17.624544 29211 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: Bootstrap starting.
W20240501 08:11:17.649837 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:11:17.658632 29211 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:17.658985 29211 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: Bootstrap complete.
I20240501 08:11:17.659252 29211 ts_tablet_manager.cc:1367] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: Time spent bootstrapping tablet: real 0.035s user 0.036s sys 0.000s
I20240501 08:11:17.659963 29211 raft_consensus.cc:348] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:17.660158 29211 raft_consensus.cc:729] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: fba0b24a97a84f6fa331664028231d19, State: Initialized, Role: LEARNER
I20240501 08:11:17.660356 29211 consensus_queue.cc:260] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:17.661063 29211 ts_tablet_manager.cc:1398] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:17.661928 26362 tablet_copy_service.cc:342] P bc13665cb8894b1abf18738f527ab244: Request end of tablet copy session fba0b24a97a84f6fa331664028231d19-359983c3db9a43be92e135c4b722dc0f received from {username='slave'} at 127.25.107.7:54191
I20240501 08:11:17.662082 26362 tablet_copy_service.cc:433] P bc13665cb8894b1abf18738f527ab244: ending tablet copy session fba0b24a97a84f6fa331664028231d19-359983c3db9a43be92e135c4b722dc0f on tablet 359983c3db9a43be92e135c4b722dc0f with peer fba0b24a97a84f6fa331664028231d19
W20240501 08:11:17.672756 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:11:17.708477 27835 raft_consensus.cc:1203] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240501 08:11:17.864267 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:11:18.063760 29359 raft_consensus.cc:1050] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244: attempting to promote NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 to VOTER
I20240501 08:11:18.064738 29359 consensus_queue.cc:237] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:18.068413 28095 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:18.068894 27703 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 LEARNER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:18.069033 26471 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:18.069641 29359 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:18.070168 29241 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:18.070822 29360 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:18.074929 29241 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:18.075559 28095 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:18.076232 27703 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:18.079406 26471 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:18.080761 26067 catalog_manager.cc:5484] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786 reported cstate change: config changed from index 8 to 9, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:18.093564 26341 consensus_queue.cc:237] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:18.095450 27703 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:11:18.095563 28095 raft_consensus.cc:1261] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:11:18.096043 29360 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:18.096369 29359 consensus_queue.cc:1035] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:18.098734 29241 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P bc13665cb8894b1abf18738f527ab244 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:18.099431 28095 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:18.099345 27703 raft_consensus.cc:2932] T 9981240ef4c1466cb37895e24a0c6e5c P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:18.102947 26055 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9981240ef4c1466cb37895e24a0c6e5c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:18.103880 26063 catalog_manager.cc:5484] T 9981240ef4c1466cb37895e24a0c6e5c P 25e18e28f3914e45be7093bf5bf436e5 reported cstate change: config changed from index 9 to 10, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:18.107847 26450 tablet_service.cc:1514] Processing DeleteTablet for tablet 9981240ef4c1466cb37895e24a0c6e5c with delete_type TABLET_DATA_TOMBSTONED (TS 875f11971b814f2ea7ce89de1e4f4587 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49828
I20240501 08:11:18.108734 29376 tablet_replica.cc:330] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587: stopping tablet replica
I20240501 08:11:18.109098 29376 raft_consensus.cc:2229] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:18.109452 29376 raft_consensus.cc:2258] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:18.110872 29376 ts_tablet_manager.cc:1875] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:18.115613 29376 ts_tablet_manager.cc:1888] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 08:11:18.115765 29376 log.cc:1196] T 9981240ef4c1466cb37895e24a0c6e5c P 875f11971b814f2ea7ce89de1e4f4587: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/9981240ef4c1466cb37895e24a0c6e5c
I20240501 08:11:18.116550 26058 catalog_manager.cc:4830] TS 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015): tablet 9981240ef4c1466cb37895e24a0c6e5c (table rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]) successfully deleted
I20240501 08:11:18.121665 29378 raft_consensus.cc:1050] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: attempting to promote NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b to VOTER
I20240501 08:11:18.122588 29378 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:18.125205 26341 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:18.125306 27835 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:18.125383 27965 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:18.125926 29210 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:18.126286 29378 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:18.126727 29380 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:18.130473 29210 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:18.131347 27835 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:18.132287 26341 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:18.135152 27965 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:18.136327 26063 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index 8 to 9, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:18.255682 27965 raft_consensus.cc:1203] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240501 08:11:18.306810 26600 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d"
dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59308
I20240501 08:11:18.307066 26600 raft_consensus.cc:595] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:18.332310 26600 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d"
dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59320
I20240501 08:11:18.332553 26600 raft_consensus.cc:595] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Received request to transfer leadership
W20240501 08:11:18.348953 26058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240501 08:11:18.349692 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:11:18.350200 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 12641e2d23ac450f9f04e41bb583b32d: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:18.557324 26600 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d"
dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59342
I20240501 08:11:18.557536 26600 raft_consensus.cc:595] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:18.593580 29377 raft_consensus.cc:979] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: : Instructing follower fba0b24a97a84f6fa331664028231d19 to start an election
I20240501 08:11:18.593739 29377 raft_consensus.cc:1067] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Signalling peer fba0b24a97a84f6fa331664028231d19 to start an election
I20240501 08:11:18.594547 27965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d"
dest_uuid: "fba0b24a97a84f6fa331664028231d19"
from {username='slave'} at 127.25.107.4:38927
I20240501 08:11:18.594807 27965 raft_consensus.cc:482] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:18.594967 27965 raft_consensus.cc:3037] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:18.596853 27965 raft_consensus.cc:504] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:18.599051 26341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d" candidate_uuid: "fba0b24a97a84f6fa331664028231d19" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:11:18.599161 26600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d" candidate_uuid: "fba0b24a97a84f6fa331664028231d19" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "f0f978a65b1549689c460fce4699e7a4"
I20240501 08:11:18.599278 26341 raft_consensus.cc:3037] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:18.599409 26600 raft_consensus.cc:3032] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:18.599491 26600 raft_consensus.cc:729] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f0f978a65b1549689c460fce4699e7a4, State: Running, Role: LEADER
I20240501 08:11:18.599797 27965 leader_election.cc:288] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [CANDIDATE]: Term 2 election: Requested vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061), f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6:34341)
I20240501 08:11:18.599841 26600 consensus_queue.cc:260] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:18.600265 26600 raft_consensus.cc:3037] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:18.602170 26341 raft_consensus.cc:2454] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fba0b24a97a84f6fa331664028231d19 in term 2.
I20240501 08:11:18.605001 26600 raft_consensus.cc:2454] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fba0b24a97a84f6fa331664028231d19 in term 2.
I20240501 08:11:18.605443 27898 leader_election.cc:302] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: bc13665cb8894b1abf18738f527ab244, f0f978a65b1549689c460fce4699e7a4, fba0b24a97a84f6fa331664028231d19; no voters:
I20240501 08:11:18.605782 29333 raft_consensus.cc:2781] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:18.606177 29333 raft_consensus.cc:686] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 2 LEADER]: Becoming Leader. State: Replica: fba0b24a97a84f6fa331664028231d19, State: Running, Role: LEADER
I20240501 08:11:18.606629 29333 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:18.610035 26069 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 reported cstate change: term changed from 1 to 2, leader changed from f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) to fba0b24a97a84f6fa331664028231d19 (127.25.107.7). New cstate: current_term: 2 leader_uuid: "fba0b24a97a84f6fa331664028231d19" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:18.619783 27835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12641e2d23ac450f9f04e41bb583b32d" candidate_uuid: "fba0b24a97a84f6fa331664028231d19" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b"
I20240501 08:11:18.620142 27835 raft_consensus.cc:3037] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:18.623243 27835 raft_consensus.cc:2454] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fba0b24a97a84f6fa331664028231d19 in term 2.
I20240501 08:11:18.701973 29241 raft_consensus.cc:1050] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: attempting to promote NON_VOTER fba0b24a97a84f6fa331664028231d19 to VOTER
I20240501 08:11:18.703255 29241 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:18.705637 27965 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 2 LEARNER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:18.706111 28095 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:18.706707 29441 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:18.707226 29441 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:18.707741 26471 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:18.711866 29444 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:18.721367 29442 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:18.721915 28095 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:18.725387 26471 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:18.727504 26069 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 9 to 10, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:18.732332 27965 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:18.776425 26341 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f"
dest_uuid: "bc13665cb8894b1abf18738f527ab244"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52008
I20240501 08:11:18.776685 26341 raft_consensus.cc:595] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Received request to transfer leadership
I20240501 08:11:18.994783 26602 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:18.995489 29333 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:19.006801 26341 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f"
dest_uuid: "bc13665cb8894b1abf18738f527ab244"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52036
I20240501 08:11:19.007000 26341 raft_consensus.cc:595] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Received request to transfer leadership
I20240501 08:11:19.104676 27835 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:19.105302 29333 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:19.105751 29241 raft_consensus.cc:979] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: : Instructing follower 875f11971b814f2ea7ce89de1e4f4587 to start an election
I20240501 08:11:19.105970 29444 raft_consensus.cc:1067] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Signalling peer 875f11971b814f2ea7ce89de1e4f4587 to start an election
I20240501 08:11:19.106920 26471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f"
dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
from {username='slave'} at 127.25.107.2:46337
I20240501 08:11:19.107138 26471 raft_consensus.cc:482] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:19.107275 26471 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:11:19.110363 26471 raft_consensus.cc:504] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:19.111979 26341 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:19.111960 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "bc13665cb8894b1abf18738f527ab244"
I20240501 08:11:19.112172 26342 raft_consensus.cc:3032] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:11:19.112257 26342 raft_consensus.cc:729] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Running, Role: LEADER
I20240501 08:11:19.112594 29485 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:19.112569 26342 consensus_queue.cc:260] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:19.113003 26342 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:11:19.115971 26342 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 3.
I20240501 08:11:19.118269 26471 leader_election.cc:288] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 3 election: Requested vote from peers bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319), 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8:33359), fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815)
I20240501 08:11:19.118433 27965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "fba0b24a97a84f6fa331664028231d19"
I20240501 08:11:19.118707 27965 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:11:19.121906 27965 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 3.
I20240501 08:11:19.122829 26408 leader_election.cc:302] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244, fba0b24a97a84f6fa331664028231d19; no voters:
I20240501 08:11:19.123108 29207 raft_consensus.cc:2781] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:11:19.123239 29207 raft_consensus.cc:686] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 LEADER]: Becoming Leader. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Running, Role: LEADER
I20240501 08:11:19.123538 29207 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:19.126106 26063 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: term changed from 2 to 3, leader changed from bc13665cb8894b1abf18738f527ab244 (127.25.107.2) to 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3). New cstate: current_term: 3 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:19.131294 28095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "875f11971b814f2ea7ce89de1e4f4587" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "25e18e28f3914e45be7093bf5bf436e5"
I20240501 08:11:19.131645 28095 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:11:19.134627 28095 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875f11971b814f2ea7ce89de1e4f4587 in term 3.
I20240501 08:11:19.525930 28095 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 3 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 08:11:19.526494 29207 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:11:19.614858 26341 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 3 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 08:11:19.615455 29207 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:11:19.618963 27965 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 3 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 08:11:19.619639 29515 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 08:11:19.678243 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:11:19.681869 27965 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:19.683705 26341 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:11:19.683705 27835 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:11:19.684234 29529 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:11:19.684557 29528 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:11:19.686750 29529 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:19.687386 27835 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:19.687744 26341 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:19.692700 26063 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 9 to 11, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New cstate: current_term: 2 leader_uuid: "fba0b24a97a84f6fa331664028231d19" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:19.693907 26058 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12641e2d23ac450f9f04e41bb583b32d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:19.697850 26582 tablet_service.cc:1514] Processing DeleteTablet for tablet 12641e2d23ac450f9f04e41bb583b32d with delete_type TABLET_DATA_TOMBSTONED (TS f0f978a65b1549689c460fce4699e7a4 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44112
I20240501 08:11:19.699328 29536 tablet_replica.cc:330] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: stopping tablet replica
I20240501 08:11:19.699888 29536 raft_consensus.cc:2229] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:19.700237 29536 raft_consensus.cc:2258] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:19.701865 29536 ts_tablet_manager.cc:1875] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:19.707492 29536 ts_tablet_manager.cc:1888] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 08:11:19.707613 29536 log.cc:1196] T 12641e2d23ac450f9f04e41bb583b32d P f0f978a65b1549689c460fce4699e7a4: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/12641e2d23ac450f9f04e41bb583b32d
I20240501 08:11:19.708344 26055 catalog_manager.cc:4830] TS f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061): tablet 12641e2d23ac450f9f04e41bb583b32d (table rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]) successfully deleted
W20240501 08:11:19.742835 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:11:19.809830 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 359983c3db9a43be92e135c4b722dc0f: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:19.903267 26471 consensus_queue.cc:237] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:19.905845 27965 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:19.905862 26341 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:19.906477 29465 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:19.906839 29515 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:19.909646 29465 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:19.910427 26341 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
W20240501 08:11:19.914446 26406 consensus_peers.cc:485] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6:34341): Couldn't send request to peer f2a0e97dec014a17a4c96bfd2f60a96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 595048c2a092461b979fe7e107882d75. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:19.915462 26069 catalog_manager.cc:5484] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 7 to 8, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:19.917893 27965 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:19.930471 26341 consensus_queue.cc:237] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:19.932688 28095 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:11:19.932904 26602 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240501 08:11:19.933033 26276 consensus_peers.cc:485] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 -> Peer 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5:43333): Couldn't send request to peer 0c0ceab7acf1424aa23a96d23e0aa786. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5b85279a473435fb21f7e6c3cbebea2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:19.933413 29444 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:19.933723 29566 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:19.936201 29558 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:19.936734 26602 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:19.936719 28095 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:19.941026 26062 catalog_manager.cc:5484] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index 8 to 9, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:20.007319 26471 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:20.009045 27965 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 3 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 08:11:20.009131 28095 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 3 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 08:11:20.009660 29465 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:11:20.010010 29207 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:11:20.012168 29515 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:20.012775 28095 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:20.012960 27965 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:20.016485 26058 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 359983c3db9a43be92e135c4b722dc0f with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:20.017732 26063 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 reported cstate change: config changed from index 10 to 12, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New cstate: current_term: 3 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:20.020987 26322 tablet_service.cc:1514] Processing DeleteTablet for tablet 359983c3db9a43be92e135c4b722dc0f with delete_type TABLET_DATA_TOMBSTONED (TS bc13665cb8894b1abf18738f527ab244 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:33068
I20240501 08:11:20.021821 29584 tablet_replica.cc:330] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: stopping tablet replica
I20240501 08:11:20.022310 29584 raft_consensus.cc:2229] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:20.022653 29584 raft_consensus.cc:2258] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:20.024063 29584 ts_tablet_manager.cc:1875] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:20.028620 29584 ts_tablet_manager.cc:1888] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240501 08:11:20.028739 29584 log.cc:1196] T 359983c3db9a43be92e135c4b722dc0f P bc13665cb8894b1abf18738f527ab244: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/359983c3db9a43be92e135c4b722dc0f
I20240501 08:11:20.029560 26055 catalog_manager.cc:4830] TS bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319): tablet 359983c3db9a43be92e135c4b722dc0f (table rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]) successfully deleted
W20240501 08:11:20.146528 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:11:20.166700 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:11:20.205286 26471 consensus_queue.cc:237] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
W20240501 08:11:20.207702 26408 consensus_peers.cc:485] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815): Couldn't send request to peer fba0b24a97a84f6fa331664028231d19. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e97904ef0a5e41cab9054cbc84aae923. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:20.208045 26341 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:11:20.208057 26602 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:11:20.208516 29207 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:20.208789 29465 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:20.211167 29515 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:20.211736 26602 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:20.214532 26341 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:20.216763 26062 catalog_manager.cc:5484] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index -1 to 4, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
W20240501 08:11:20.238466 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:11:20.319514 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:11:20.348325 29625 ts_tablet_manager.cc:902] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: Initiating tablet copy from peer bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319)
I20240501 08:11:20.349510 29625 tablet_copy_client.cc:308] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:34319
I20240501 08:11:20.350169 26362 tablet_copy_service.cc:140] P bc13665cb8894b1abf18738f527ab244: Received BeginTabletCopySession request for tablet a5b85279a473435fb21f7e6c3cbebea2 from peer 0c0ceab7acf1424aa23a96d23e0aa786 ({username='slave'} at 127.25.107.5:40265)
I20240501 08:11:20.350358 26362 tablet_copy_service.cc:161] P bc13665cb8894b1abf18738f527ab244: Beginning new tablet copy session on tablet a5b85279a473435fb21f7e6c3cbebea2 from peer 0c0ceab7acf1424aa23a96d23e0aa786 at {username='slave'} at 127.25.107.5:40265: session id = 0c0ceab7acf1424aa23a96d23e0aa786-a5b85279a473435fb21f7e6c3cbebea2
I20240501 08:11:20.352242 26362 tablet_copy_source_session.cc:205] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:20.353374 29625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b85279a473435fb21f7e6c3cbebea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:20.357583 29625 tablet_copy_client.cc:783] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:20.357795 29625 tablet_copy_client.cc:647] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:20.359253 29629 tablet_copy_client.cc:1002] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Injecting latency into file download: 1500
I20240501 08:11:20.362282 29630 ts_tablet_manager.cc:902] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: Initiating tablet copy from peer 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:11:20.363642 29630 tablet_copy_client.cc:308] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:41015
I20240501 08:11:20.370286 26492 tablet_copy_service.cc:140] P 875f11971b814f2ea7ce89de1e4f4587: Received BeginTabletCopySession request for tablet 595048c2a092461b979fe7e107882d75 from peer f2a0e97dec014a17a4c96bfd2f60a96b ({username='slave'} at 127.25.107.6:57311)
I20240501 08:11:20.370477 26492 tablet_copy_service.cc:161] P 875f11971b814f2ea7ce89de1e4f4587: Beginning new tablet copy session on tablet 595048c2a092461b979fe7e107882d75 from peer f2a0e97dec014a17a4c96bfd2f60a96b at {username='slave'} at 127.25.107.6:57311: session id = f2a0e97dec014a17a4c96bfd2f60a96b-595048c2a092461b979fe7e107882d75
I20240501 08:11:20.373550 26492 tablet_copy_source_session.cc:205] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:20.374883 29630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595048c2a092461b979fe7e107882d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:20.381297 29630 tablet_copy_client.cc:783] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:20.381531 29630 tablet_copy_client.cc:647] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:20.383085 29634 tablet_copy_client.cc:1002] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Injecting latency into file download: 1500
I20240501 08:11:20.660786 29651 ts_tablet_manager.cc:902] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: Initiating tablet copy from peer 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:11:20.661932 29651 tablet_copy_client.cc:308] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:41015
I20240501 08:11:20.662621 26492 tablet_copy_service.cc:140] P 875f11971b814f2ea7ce89de1e4f4587: Received BeginTabletCopySession request for tablet e97904ef0a5e41cab9054cbc84aae923 from peer fba0b24a97a84f6fa331664028231d19 ({username='slave'} at 127.25.107.7:47561)
I20240501 08:11:20.662775 26492 tablet_copy_service.cc:161] P 875f11971b814f2ea7ce89de1e4f4587: Beginning new tablet copy session on tablet e97904ef0a5e41cab9054cbc84aae923 from peer fba0b24a97a84f6fa331664028231d19 at {username='slave'} at 127.25.107.7:47561: session id = fba0b24a97a84f6fa331664028231d19-e97904ef0a5e41cab9054cbc84aae923
I20240501 08:11:20.664650 26492 tablet_copy_source_session.cc:205] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:20.665740 29651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97904ef0a5e41cab9054cbc84aae923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:20.669802 29651 tablet_copy_client.cc:783] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:20.670019 29651 tablet_copy_client.cc:647] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:20.671309 29655 tablet_copy_client.cc:1002] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: tablet copy: Injecting latency into file download: 1500
W20240501 08:11:20.776913 26538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 08:11:20.864771 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:11:21.071398 26408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:11:21.324438 26278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:11:21.633397 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9981240ef4c1466cb37895e24a0c6e5c: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:21.859599 29625 tablet_copy_client.cc:523] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:21.862454 29625 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap starting.
I20240501 08:11:21.883487 29630 tablet_copy_client.cc:523] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:21.887110 29630 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap starting.
I20240501 08:11:21.900007 29625 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:21.900393 29625 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap complete.
I20240501 08:11:21.900653 29625 ts_tablet_manager.cc:1367] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: Time spent bootstrapping tablet: real 0.038s user 0.026s sys 0.010s
I20240501 08:11:21.901607 29625 raft_consensus.cc:348] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:21.901907 29625 raft_consensus.cc:729] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0c0ceab7acf1424aa23a96d23e0aa786, State: Initialized, Role: LEARNER
I20240501 08:11:21.902114 29625 consensus_queue.cc:260] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:21.903091 29625 ts_tablet_manager.cc:1398] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 08:11:21.903898 26362 tablet_copy_service.cc:342] P bc13665cb8894b1abf18738f527ab244: Request end of tablet copy session 0c0ceab7acf1424aa23a96d23e0aa786-a5b85279a473435fb21f7e6c3cbebea2 received from {username='slave'} at 127.25.107.5:40265
I20240501 08:11:21.904042 26362 tablet_copy_service.cc:433] P bc13665cb8894b1abf18738f527ab244: ending tablet copy session 0c0ceab7acf1424aa23a96d23e0aa786-a5b85279a473435fb21f7e6c3cbebea2 on tablet a5b85279a473435fb21f7e6c3cbebea2 with peer 0c0ceab7acf1424aa23a96d23e0aa786
I20240501 08:11:21.923604 29630 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:21.923991 29630 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap complete.
I20240501 08:11:21.924222 29630 ts_tablet_manager.cc:1367] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.005s
I20240501 08:11:21.924968 29630 raft_consensus.cc:348] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:21.925233 29630 raft_consensus.cc:729] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f2a0e97dec014a17a4c96bfd2f60a96b, State: Initialized, Role: LEARNER
I20240501 08:11:21.925442 29630 consensus_queue.cc:260] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:21.926199 29630 ts_tablet_manager.cc:1398] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240501 08:11:21.926966 26492 tablet_copy_service.cc:342] P 875f11971b814f2ea7ce89de1e4f4587: Request end of tablet copy session f2a0e97dec014a17a4c96bfd2f60a96b-595048c2a092461b979fe7e107882d75 received from {username='slave'} at 127.25.107.6:57311
I20240501 08:11:21.927114 26492 tablet_copy_service.cc:433] P 875f11971b814f2ea7ce89de1e4f4587: ending tablet copy session f2a0e97dec014a17a4c96bfd2f60a96b-595048c2a092461b979fe7e107882d75 on tablet 595048c2a092461b979fe7e107882d75 with peer f2a0e97dec014a17a4c96bfd2f60a96b
I20240501 08:11:21.976112 27703 raft_consensus.cc:1203] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240501 08:11:22.106997 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:11:22.171720 29651 tablet_copy_client.cc:523] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:22.174382 29651 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: Bootstrap starting.
I20240501 08:11:22.203146 29651 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:22.203521 29651 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: Bootstrap complete.
I20240501 08:11:22.203734 29651 ts_tablet_manager.cc:1367] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.005s
I20240501 08:11:22.204387 29651 raft_consensus.cc:348] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:22.204622 29651 raft_consensus.cc:729] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fba0b24a97a84f6fa331664028231d19, State: Initialized, Role: LEARNER
I20240501 08:11:22.204794 29651 consensus_queue.cc:260] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:22.205240 29651 ts_tablet_manager.cc:1398] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:22.206022 26492 tablet_copy_service.cc:342] P 875f11971b814f2ea7ce89de1e4f4587: Request end of tablet copy session fba0b24a97a84f6fa331664028231d19-e97904ef0a5e41cab9054cbc84aae923 received from {username='slave'} at 127.25.107.7:47561
I20240501 08:11:22.206174 26492 tablet_copy_service.cc:433] P 875f11971b814f2ea7ce89de1e4f4587: ending tablet copy session fba0b24a97a84f6fa331664028231d19-e97904ef0a5e41cab9054cbc84aae923 on tablet e97904ef0a5e41cab9054cbc84aae923 with peer fba0b24a97a84f6fa331664028231d19
W20240501 08:11:22.245048 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:11:22.449391 27835 raft_consensus.cc:1203] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 08:11:22.456238 29798 raft_consensus.cc:1050] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244: attempting to promote NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 to VOTER
I20240501 08:11:22.456876 29798 consensus_queue.cc:237] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:22.459268 28095 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:22.459447 27703 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 LEARNER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:22.459836 29725 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:22.460012 26600 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:22.460193 29797 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:22.460546 29798 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:22.463704 29797 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:22.464443 27703 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:22.464815 28095 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:22.469377 26069 catalog_manager.cc:5484] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786 reported cstate change: config changed from index 9 to 10, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:22.470755 26600 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:22.479919 26341 consensus_queue.cc:237] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:22.482010 28095 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:11:22.482012 27703 raft_consensus.cc:1261] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:11:22.482504 29798 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:11:22.483007 29798 consensus_queue.cc:1035] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:11:22.484653 29798 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:22.485476 28095 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P 25e18e28f3914e45be7093bf5bf436e5 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:22.485854 27703 raft_consensus.cc:2932] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:22.488394 26055 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a5b85279a473435fb21f7e6c3cbebea2 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:22.490070 26063 catalog_manager.cc:5484] T a5b85279a473435fb21f7e6c3cbebea2 P 0c0ceab7acf1424aa23a96d23e0aa786 reported cstate change: config changed from index 10 to 11, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:22.493713 26582 tablet_service.cc:1514] Processing DeleteTablet for tablet a5b85279a473435fb21f7e6c3cbebea2 with delete_type TABLET_DATA_TOMBSTONED (TS f0f978a65b1549689c460fce4699e7a4 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44112
I20240501 08:11:22.498760 29810 tablet_replica.cc:330] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4: stopping tablet replica
I20240501 08:11:22.499029 29810 raft_consensus.cc:2229] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:22.499408 29810 raft_consensus.cc:2258] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:22.500258 29810 ts_tablet_manager.cc:1875] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:22.505789 29810 ts_tablet_manager.cc:1888] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 08:11:22.505928 29810 log.cc:1196] T a5b85279a473435fb21f7e6c3cbebea2 P f0f978a65b1549689c460fce4699e7a4: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/a5b85279a473435fb21f7e6c3cbebea2
I20240501 08:11:22.506740 26055 catalog_manager.cc:4830] TS f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061): tablet a5b85279a473435fb21f7e6c3cbebea2 (table rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]) successfully deleted
W20240501 08:11:22.705590 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:11:22.707930 27965 raft_consensus.cc:1203] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 08:11:22.745986 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:11:22.761143 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:11:22.893381 29845 raft_consensus.cc:1050] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: attempting to promote NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b to VOTER
I20240501 08:11:22.894379 29845 consensus_queue.cc:237] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:22.896476 27835 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:22.896505 27965 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:22.896936 26341 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:22.897078 29845 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:22.897501 29677 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:22.898028 29844 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:22.900967 29845 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:22.901463 27835 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:22.906898 26063 catalog_manager.cc:5484] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 8 to 9, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:22.906980 26341 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:22.908193 27965 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
W20240501 08:11:22.918741 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:11:23.010532 26471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "595048c2a092461b979fe7e107882d75"
dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33390
I20240501 08:11:23.010813 26471 raft_consensus.cc:595] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Received request to transfer leadership
W20240501 08:11:23.133419 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 595048c2a092461b979fe7e107882d75: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:23.235731 26471 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "595048c2a092461b979fe7e107882d75"
dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33422
I20240501 08:11:23.235900 26471 raft_consensus.cc:595] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:23.257160 29851 raft_consensus.cc:1050] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587: attempting to promote NON_VOTER fba0b24a97a84f6fa331664028231d19 to VOTER
I20240501 08:11:23.258076 29851 consensus_queue.cc:237] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:23.260666 27965 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:23.260713 26341 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:23.260830 26600 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:23.261310 29677 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:23.261665 29851 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:23.261957 29850 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:23.264963 29677 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:23.265571 26600 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:23.265733 27965 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:23.270053 26062 catalog_manager.cc:5484] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index 4 to 5, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:23.270947 26341 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:23.277482 26471 consensus_queue.cc:237] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:23.279356 26600 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:23.279397 27965 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:23.279994 29845 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:23.280426 29844 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:23.281920 29845 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:23.282740 26600 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:23.285750 26058 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e97904ef0a5e41cab9054cbc84aae923 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:23.286128 27965 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:23.287439 26067 catalog_manager.cc:5484] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:23.291007 26322 tablet_service.cc:1514] Processing DeleteTablet for tablet e97904ef0a5e41cab9054cbc84aae923 with delete_type TABLET_DATA_TOMBSTONED (TS bc13665cb8894b1abf18738f527ab244 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33068
I20240501 08:11:23.291672 29888 tablet_replica.cc:330] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244: stopping tablet replica
I20240501 08:11:23.291862 29888 raft_consensus.cc:2229] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:23.292181 29888 raft_consensus.cc:2258] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:23.292960 29888 ts_tablet_manager.cc:1875] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:23.297174 29888 ts_tablet_manager.cc:1888] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:11:23.297307 29888 log.cc:1196] T e97904ef0a5e41cab9054cbc84aae923 P bc13665cb8894b1abf18738f527ab244: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/e97904ef0a5e41cab9054cbc84aae923
I20240501 08:11:23.297938 26055 catalog_manager.cc:4830] TS bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319): tablet e97904ef0a5e41cab9054cbc84aae923 (table rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]) successfully deleted
I20240501 08:11:23.406240 29677 raft_consensus.cc:979] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: : Instructing follower bc13665cb8894b1abf18738f527ab244 to start an election
I20240501 08:11:23.406510 29677 raft_consensus.cc:1067] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Signalling peer bc13665cb8894b1abf18738f527ab244 to start an election
I20240501 08:11:23.407362 26341 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "595048c2a092461b979fe7e107882d75"
dest_uuid: "bc13665cb8894b1abf18738f527ab244"
from {username='slave'} at 127.25.107.3:52709
I20240501 08:11:23.407661 26341 raft_consensus.cc:482] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:23.407784 26341 raft_consensus.cc:3037] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:23.410782 26341 raft_consensus.cc:504] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:23.412575 26341 leader_election.cc:288] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 election: Requested vote from peers 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015), fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815), f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6:34341)
I20240501 08:11:23.412618 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595048c2a092461b979fe7e107882d75" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
I20240501 08:11:23.412849 26471 raft_consensus.cc:3032] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:23.412958 26471 raft_consensus.cc:729] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Running, Role: LEADER
I20240501 08:11:23.413339 27965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595048c2a092461b979fe7e107882d75" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "fba0b24a97a84f6fa331664028231d19"
I20240501 08:11:23.413308 26471 consensus_queue.cc:260] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:23.413595 27965 raft_consensus.cc:3037] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:23.413831 26471 raft_consensus.cc:3037] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:23.417093 26471 raft_consensus.cc:2454] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 2.
I20240501 08:11:23.417084 27965 raft_consensus.cc:2454] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 2.
I20240501 08:11:23.417781 26278 leader_election.cc:302] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 875f11971b814f2ea7ce89de1e4f4587, bc13665cb8894b1abf18738f527ab244, fba0b24a97a84f6fa331664028231d19; no voters:
I20240501 08:11:23.418274 29798 raft_consensus.cc:2781] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:23.418678 29798 raft_consensus.cc:686] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Becoming Leader. State: Replica: bc13665cb8894b1abf18738f527ab244, State: Running, Role: LEADER
I20240501 08:11:23.419032 29798 consensus_queue.cc:237] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:23.420112 27835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595048c2a092461b979fe7e107882d75" candidate_uuid: "bc13665cb8894b1abf18738f527ab244" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b"
I20240501 08:11:23.420333 27835 raft_consensus.cc:3037] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:23.422911 27835 raft_consensus.cc:2454] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc13665cb8894b1abf18738f527ab244 in term 2.
I20240501 08:11:23.422870 26069 catalog_manager.cc:5484] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 reported cstate change: term changed from 1 to 2, leader changed from 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) to bc13665cb8894b1abf18738f527ab244 (127.25.107.2). New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:23.454367 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:11:23.467677 26471 consensus_queue.cc:237] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:23.470050 26600 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:11:23.470307 26408 consensus_peers.cc:485] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815): Couldn't send request to peer fba0b24a97a84f6fa331664028231d19. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3ddc36d3fc146938c366df2c7f090a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:23.470326 26341 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:11:23.470671 29677 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:23.470992 29844 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:23.473825 29677 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:23.474910 26600 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:23.475242 26341 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:23.479526 26069 catalog_manager.cc:5484] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index -1 to 4, NON_VOTER fba0b24a97a84f6fa331664028231d19 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } } }
I20240501 08:11:23.870751 26471 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:23.871381 29798 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:23.943264 27835 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:23.943851 29798 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:23.948693 27965 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:11:23.949465 29945 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:11:24.001039 29950 ts_tablet_manager.cc:902] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: Initiating tablet copy from peer 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:11:24.002658 29950 tablet_copy_client.cc:308] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:41015
I20240501 08:11:24.003495 26492 tablet_copy_service.cc:140] P 875f11971b814f2ea7ce89de1e4f4587: Received BeginTabletCopySession request for tablet c3ddc36d3fc146938c366df2c7f090a6 from peer fba0b24a97a84f6fa331664028231d19 ({username='slave'} at 127.25.107.7:47561)
I20240501 08:11:24.003706 26492 tablet_copy_service.cc:161] P 875f11971b814f2ea7ce89de1e4f4587: Beginning new tablet copy session on tablet c3ddc36d3fc146938c366df2c7f090a6 from peer fba0b24a97a84f6fa331664028231d19 at {username='slave'} at 127.25.107.7:47561: session id = fba0b24a97a84f6fa331664028231d19-c3ddc36d3fc146938c366df2c7f090a6
I20240501 08:11:24.006790 26492 tablet_copy_source_session.cc:205] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:24.008150 29950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ddc36d3fc146938c366df2c7f090a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:24.013224 29950 tablet_copy_client.cc:783] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:24.013487 29950 tablet_copy_client.cc:647] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:24.015070 29954 tablet_copy_client.cc:1002] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: tablet copy: Injecting latency into file download: 1500
I20240501 08:11:24.480722 26341 consensus_queue.cc:237] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:24.482825 27965 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:11:24.482993 27835 raft_consensus.cc:1261] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Refusing update from remote peer bc13665cb8894b1abf18738f527ab244: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:11:24.483462 29979 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:11:24.483843 29980 consensus_queue.cc:1035] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:11:24.486323 29798 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:24.486996 27835 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:24.489398 26055 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 595048c2a092461b979fe7e107882d75 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:24.489763 27965 raft_consensus.cc:2932] T 595048c2a092461b979fe7e107882d75 P fba0b24a97a84f6fa331664028231d19 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:24.491415 26063 catalog_manager.cc:5484] T 595048c2a092461b979fe7e107882d75 P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 9 to 11, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New cstate: current_term: 2 leader_uuid: "bc13665cb8894b1abf18738f527ab244" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:24.498154 26450 tablet_service.cc:1514] Processing DeleteTablet for tablet 595048c2a092461b979fe7e107882d75 with delete_type TABLET_DATA_TOMBSTONED (TS 875f11971b814f2ea7ce89de1e4f4587 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49828
I20240501 08:11:24.498848 29986 tablet_replica.cc:330] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: stopping tablet replica
I20240501 08:11:24.499194 29986 raft_consensus.cc:2229] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:24.499480 29986 raft_consensus.cc:2258] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:24.501056 29986 ts_tablet_manager.cc:1875] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:24.505548 29986 ts_tablet_manager.cc:1888] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 08:11:24.505676 29986 log.cc:1196] T 595048c2a092461b979fe7e107882d75 P 875f11971b814f2ea7ce89de1e4f4587: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/595048c2a092461b979fe7e107882d75
I20240501 08:11:24.506441 26058 catalog_manager.cc:4830] TS 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015): tablet 595048c2a092461b979fe7e107882d75 (table rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]) successfully deleted
W20240501 08:11:24.530385 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:11:24.883875 26058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240501 08:11:24.885145 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a5b85279a473435fb21f7e6c3cbebea2: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:24.984167 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:11:25.059953 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:11:25.082175 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:11:25.343293 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:11:25.360775 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:11:25.515413 29950 tablet_copy_client.cc:523] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:25.518065 29950 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: Bootstrap starting.
I20240501 08:11:25.548569 29950 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:25.548944 29950 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: Bootstrap complete.
I20240501 08:11:25.549194 29950 ts_tablet_manager.cc:1367] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: Time spent bootstrapping tablet: real 0.031s user 0.029s sys 0.002s
I20240501 08:11:25.549885 29950 raft_consensus.cc:348] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:25.550086 29950 raft_consensus.cc:729] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fba0b24a97a84f6fa331664028231d19, State: Initialized, Role: LEARNER
I20240501 08:11:25.550240 29950 consensus_queue.cc:260] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:25.550709 29950 ts_tablet_manager.cc:1398] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19: Time spent starting tablet: real 0.001s user 0.001s sys 0.003s
I20240501 08:11:25.551497 26492 tablet_copy_service.cc:342] P 875f11971b814f2ea7ce89de1e4f4587: Request end of tablet copy session fba0b24a97a84f6fa331664028231d19-c3ddc36d3fc146938c366df2c7f090a6 received from {username='slave'} at 127.25.107.7:47561
I20240501 08:11:25.551648 26492 tablet_copy_service.cc:433] P 875f11971b814f2ea7ce89de1e4f4587: ending tablet copy session fba0b24a97a84f6fa331664028231d19-c3ddc36d3fc146938c366df2c7f090a6 on tablet c3ddc36d3fc146938c366df2c7f090a6 with peer fba0b24a97a84f6fa331664028231d19
I20240501 08:11:25.798540 27965 raft_consensus.cc:1203] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 08:11:25.843415 26538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 08:11:26.031497 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:11:26.318877 26408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 08:11:26.374840 30050 raft_consensus.cc:1050] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587: attempting to promote NON_VOTER fba0b24a97a84f6fa331664028231d19 to VOTER
I20240501 08:11:26.375648 30050 consensus_queue.cc:237] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:26.378129 27965 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 LEARNER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:26.378644 26341 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:26.378787 26600 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:26.378779 30052 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:26.379292 30053 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:26.379667 30050 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:11:26.384853 26278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 08:11:26.385403 29978 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:26.389716 27965 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:26.389796 26341 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:26.390295 26600 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:26.390926 26067 catalog_manager.cc:5484] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 4 to 5, fba0b24a97a84f6fa331664028231d19 (127.25.107.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:26.402797 26471 consensus_queue.cc:237] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } }
I20240501 08:11:26.404649 26600 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:26.404912 27965 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:26.405221 30053 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:26.405511 30050 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:26.408136 30050 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:26.408761 26600 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:26.412691 26058 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3ddc36d3fc146938c366df2c7f090a6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:26.413388 26062 catalog_manager.cc:5484] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:26.415539 27965 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } }
I20240501 08:11:26.420286 26322 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ddc36d3fc146938c366df2c7f090a6 with delete_type TABLET_DATA_TOMBSTONED (TS bc13665cb8894b1abf18738f527ab244 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33068
I20240501 08:11:26.422119 30063 tablet_replica.cc:330] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244: stopping tablet replica
I20240501 08:11:26.422479 30063 raft_consensus.cc:2229] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:26.422966 30063 raft_consensus.cc:2258] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:26.424360 30063 ts_tablet_manager.cc:1875] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:26.429037 30063 ts_tablet_manager.cc:1888] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:11:26.429175 30063 log.cc:1196] T c3ddc36d3fc146938c366df2c7f090a6 P bc13665cb8894b1abf18738f527ab244: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3ddc36d3fc146938c366df2c7f090a6
I20240501 08:11:26.430048 26055 catalog_manager.cc:4830] TS bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319): tablet c3ddc36d3fc146938c366df2c7f090a6 (table rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]) successfully deleted
I20240501 08:11:26.518342 26600 consensus_queue.cc:237] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:26.521132 26341 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:11:26.521206 26471 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:11:26.521651 26536 consensus_peers.cc:485] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 -> Peer f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6:34341): Couldn't send request to peer f2a0e97dec014a17a4c96bfd2f60a96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7206513860864277891e0f7159736271. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:26.521809 30077 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:26.522277 30076 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:26.525712 30077 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:26.526446 26471 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:26.526964 26341 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:26.530974 26062 catalog_manager.cc:5484] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index -1 to 4, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:27.046133 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:11:27.120352 30097 ts_tablet_manager.cc:902] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: Initiating tablet copy from peer f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:11:27.121665 30097 tablet_copy_client.cc:308] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:37061
I20240501 08:11:27.122498 26622 tablet_copy_service.cc:140] P f0f978a65b1549689c460fce4699e7a4: Received BeginTabletCopySession request for tablet 7206513860864277891e0f7159736271 from peer f2a0e97dec014a17a4c96bfd2f60a96b ({username='slave'} at 127.25.107.6:48597)
I20240501 08:11:27.122654 26622 tablet_copy_service.cc:161] P f0f978a65b1549689c460fce4699e7a4: Beginning new tablet copy session on tablet 7206513860864277891e0f7159736271 from peer f2a0e97dec014a17a4c96bfd2f60a96b at {username='slave'} at 127.25.107.6:48597: session id = f2a0e97dec014a17a4c96bfd2f60a96b-7206513860864277891e0f7159736271
I20240501 08:11:27.124645 26622 tablet_copy_source_session.cc:205] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:27.125972 30097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7206513860864277891e0f7159736271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:27.130661 30097 tablet_copy_client.cc:783] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:27.130893 30097 tablet_copy_client.cc:647] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:27.132560 30101 tablet_copy_client.cc:1002] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Injecting latency into file download: 1500
W20240501 08:11:27.287882 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:11:27.411379 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:11:27.437978 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:11:27.832078 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:11:27.922636 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:11:28.018747 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 359983c3db9a43be92e135c4b722dc0f: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:28.452629 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:11:28.632964 30097 tablet_copy_client.cc:523] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:28.636140 30097 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap starting.
I20240501 08:11:28.673427 30097 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:28.673828 30097 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap complete.
I20240501 08:11:28.674129 30097 ts_tablet_manager.cc:1367] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: Time spent bootstrapping tablet: real 0.038s user 0.028s sys 0.007s
I20240501 08:11:28.674932 30097 raft_consensus.cc:348] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:28.675134 30097 raft_consensus.cc:729] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f2a0e97dec014a17a4c96bfd2f60a96b, State: Initialized, Role: LEARNER
I20240501 08:11:28.675315 30097 consensus_queue.cc:260] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:28.675997 30097 ts_tablet_manager.cc:1398] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 08:11:28.676810 26622 tablet_copy_service.cc:342] P f0f978a65b1549689c460fce4699e7a4: Request end of tablet copy session f2a0e97dec014a17a4c96bfd2f60a96b-7206513860864277891e0f7159736271 received from {username='slave'} at 127.25.107.6:48597
I20240501 08:11:28.676961 26622 tablet_copy_service.cc:433] P f0f978a65b1549689c460fce4699e7a4: ending tablet copy session f2a0e97dec014a17a4c96bfd2f60a96b-7206513860864277891e0f7159736271 on tablet 7206513860864277891e0f7159736271 with peer f2a0e97dec014a17a4c96bfd2f60a96b
I20240501 08:11:28.711241 27835 raft_consensus.cc:1203] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:29.299503 30170 raft_consensus.cc:1050] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4: attempting to promote NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b to VOTER
I20240501 08:11:29.300402 30170 consensus_queue.cc:237] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:29.303032 27835 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:29.303283 26471 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:29.303773 30172 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:29.303969 26341 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:29.304195 30170 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:29.304538 30169 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:29.307852 30170 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:29.308490 26471 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:29.308537 26341 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:29.313335 27835 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:29.313637 26067 catalog_manager.cc:5484] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 4 to 5, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:29.322927 26600 consensus_queue.cc:237] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:29.324910 26471 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:29.325060 27835 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:11:29.325523 30139 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:29.325824 30172 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:29.328243 30169 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:29.328989 27835 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:29.331094 26471 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:29.331701 26055 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7206513860864277891e0f7159736271 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:29.334307 26069 catalog_manager.cc:5484] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b reported cstate change: config changed from index 5 to 6, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:29.338305 26322 tablet_service.cc:1514] Processing DeleteTablet for tablet 7206513860864277891e0f7159736271 with delete_type TABLET_DATA_TOMBSTONED (TS bc13665cb8894b1abf18738f527ab244 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33068
I20240501 08:11:29.338954 30181 tablet_replica.cc:330] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244: stopping tablet replica
I20240501 08:11:29.339232 30181 raft_consensus.cc:2229] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:29.339555 30181 raft_consensus.cc:2258] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:29.340452 30181 ts_tablet_manager.cc:1875] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:29.345669 30181 ts_tablet_manager.cc:1888] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:11:29.345815 30181 log.cc:1196] T 7206513860864277891e0f7159736271 P bc13665cb8894b1abf18738f527ab244: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/7206513860864277891e0f7159736271
I20240501 08:11:29.346568 26055 catalog_manager.cc:4830] TS bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319): tablet 7206513860864277891e0f7159736271 (table rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]) successfully deleted
I20240501 08:11:29.606199 26450 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:29.606431 28075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:29.625429 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:29.640813 26322 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:29.642184 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:29.646545 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 08:11:29.651345 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:11:29.662547 27945 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:11:29.708948 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:11:29.742946 26600 consensus_queue.cc:237] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:29.745306 27835 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:11:29.745486 26471 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:11:29.745932 30170 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:29.746341 30139 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:29.748940 30172 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:29.750003 27835 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
W20240501 08:11:29.752861 26536 consensus_peers.cc:485] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 -> Peer 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5:43333): Couldn't send request to peer 0c0ceab7acf1424aa23a96d23e0aa786. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7206513860864277891e0f7159736271. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:29.753371 26471 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:29.755254 26062 catalog_manager.cc:5484] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b reported cstate change: config changed from index 6 to 7, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:29.770797 26472 consensus_queue.cc:237] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:29.773384 27965 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:11:29.773619 26406 consensus_peers.cc:485] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6:34341): Couldn't send request to peer f2a0e97dec014a17a4c96bfd2f60a96b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3ddc36d3fc146938c366df2c7f090a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:29.774065 30050 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:29.777278 30050 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:29.777680 27965 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:29.783514 26062 catalog_manager.cc:5484] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 reported cstate change: config changed from index 6 to 7, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:29.787356 26600 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:11:29.787989 30238 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:29.789837 26600 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } } }
I20240501 08:11:29.886878 30240 consensus_queue.cc:579] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Leader has been unable to successfully communicate with peer c0624af4925d4686b5896f487f864943 for more than 30 seconds (30.033s)
W20240501 08:11:29.888696 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:11:30.041183 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:11:30.062638 30050 consensus_queue.cc:579] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Leader has been unable to successfully communicate with peer c0624af4925d4686b5896f487f864943 for more than 30 seconds (30.160s)
W20240501 08:11:30.064590 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:11:30.129449 30172 consensus_queue.cc:579] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Leader has been unable to successfully communicate with peer c0624af4925d4686b5896f487f864943 for more than 30 seconds (30.088s)
I20240501 08:11:30.156625 30262 ts_tablet_manager.cc:902] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: Initiating tablet copy from peer 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:11:30.157864 30262 tablet_copy_client.cc:308] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:41015
I20240501 08:11:30.158681 26492 tablet_copy_service.cc:140] P 875f11971b814f2ea7ce89de1e4f4587: Received BeginTabletCopySession request for tablet c3ddc36d3fc146938c366df2c7f090a6 from peer f2a0e97dec014a17a4c96bfd2f60a96b ({username='slave'} at 127.25.107.6:57311)
I20240501 08:11:30.158845 26492 tablet_copy_service.cc:161] P 875f11971b814f2ea7ce89de1e4f4587: Beginning new tablet copy session on tablet c3ddc36d3fc146938c366df2c7f090a6 from peer f2a0e97dec014a17a4c96bfd2f60a96b at {username='slave'} at 127.25.107.6:57311: session id = f2a0e97dec014a17a4c96bfd2f60a96b-c3ddc36d3fc146938c366df2c7f090a6
I20240501 08:11:30.161217 26492 tablet_copy_source_session.cc:205] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:30.162822 30262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ddc36d3fc146938c366df2c7f090a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:30.167474 30262 tablet_copy_client.cc:783] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:30.167816 30262 tablet_copy_client.cc:647] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:30.169469 30266 tablet_copy_client.cc:1002] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Injecting latency into file download: 1500
I20240501 08:11:30.240752 30272 ts_tablet_manager.cc:902] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: Initiating tablet copy from peer f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061)
I20240501 08:11:30.242028 30272 tablet_copy_client.cc:308] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:37061
I20240501 08:11:30.249040 26622 tablet_copy_service.cc:140] P f0f978a65b1549689c460fce4699e7a4: Received BeginTabletCopySession request for tablet 7206513860864277891e0f7159736271 from peer 0c0ceab7acf1424aa23a96d23e0aa786 ({username='slave'} at 127.25.107.5:56199)
I20240501 08:11:30.249265 26622 tablet_copy_service.cc:161] P f0f978a65b1549689c460fce4699e7a4: Beginning new tablet copy session on tablet 7206513860864277891e0f7159736271 from peer 0c0ceab7acf1424aa23a96d23e0aa786 at {username='slave'} at 127.25.107.5:56199: session id = 0c0ceab7acf1424aa23a96d23e0aa786-7206513860864277891e0f7159736271
I20240501 08:11:30.252007 26622 tablet_copy_source_session.cc:205] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:30.253525 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7206513860864277891e0f7159736271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:30.259533 30272 tablet_copy_client.cc:783] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:30.259902 30272 tablet_copy_client.cc:647] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:30.261544 30276 tablet_copy_client.cc:1002] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Injecting latency into file download: 1500
I20240501 08:11:30.360018 29980 consensus_queue.cc:579] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 [LEADER]: Leader has been unable to successfully communicate with peer c0624af4925d4686b5896f487f864943 for more than 30 seconds (30.477s)
W20240501 08:11:30.362290 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:11:30.481563 30172 consensus_queue.cc:579] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Leader has been unable to successfully communicate with peer c0624af4925d4686b5896f487f864943 for more than 30 seconds (30.385s)
W20240501 08:11:30.996142 26538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:11:30.996975 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:11:31.368096 26058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:11:31.369194 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 595048c2a092461b979fe7e107882d75: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:31.375046 26408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:11:31.397503 26278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 08:11:31.669924 30262 tablet_copy_client.cc:523] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:31.672777 30262 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap starting.
I20240501 08:11:31.680666 30172 consensus_queue.cc:579] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Leader has been unable to successfully communicate with peer c0624af4925d4686b5896f487f864943 for more than 30 seconds (30.198s)
I20240501 08:11:31.705384 30262 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:31.705752 30262 tablet_bootstrap.cc:492] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: Bootstrap complete.
I20240501 08:11:31.705998 30262 ts_tablet_manager.cc:1367] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.008s
I20240501 08:11:31.706661 30262 raft_consensus.cc:348] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:31.706872 30262 raft_consensus.cc:729] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f2a0e97dec014a17a4c96bfd2f60a96b, State: Initialized, Role: LEARNER
I20240501 08:11:31.707051 30262 consensus_queue.cc:260] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: NON_VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: true } }
I20240501 08:11:31.707749 30262 ts_tablet_manager.cc:1398] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20240501 08:11:31.708511 26492 tablet_copy_service.cc:342] P 875f11971b814f2ea7ce89de1e4f4587: Request end of tablet copy session f2a0e97dec014a17a4c96bfd2f60a96b-c3ddc36d3fc146938c366df2c7f090a6 received from {username='slave'} at 127.25.107.6:57311
I20240501 08:11:31.708642 26492 tablet_copy_service.cc:433] P 875f11971b814f2ea7ce89de1e4f4587: ending tablet copy session f2a0e97dec014a17a4c96bfd2f60a96b-c3ddc36d3fc146938c366df2c7f090a6 on tablet c3ddc36d3fc146938c366df2c7f090a6 with peer f2a0e97dec014a17a4c96bfd2f60a96b
I20240501 08:11:31.726634 30240 consensus_queue.cc:579] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Leader has been unable to successfully communicate with peer c0624af4925d4686b5896f487f864943 for more than 30 seconds (30.039s)
I20240501 08:11:31.762039 30272 tablet_copy_client.cc:523] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:31.764904 30272 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap starting.
I20240501 08:11:31.808809 30272 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:31.809218 30272 tablet_bootstrap.cc:492] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap complete.
I20240501 08:11:31.809453 30272 ts_tablet_manager.cc:1367] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: Time spent bootstrapping tablet: real 0.045s user 0.033s sys 0.011s
I20240501 08:11:31.810132 30272 raft_consensus.cc:348] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:31.810398 30272 raft_consensus.cc:729] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0c0ceab7acf1424aa23a96d23e0aa786, State: Initialized, Role: LEARNER
I20240501 08:11:31.810576 30272 consensus_queue.cc:260] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:31.811251 30272 ts_tablet_manager.cc:1398] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:11:31.812072 26622 tablet_copy_service.cc:342] P f0f978a65b1549689c460fce4699e7a4: Request end of tablet copy session 0c0ceab7acf1424aa23a96d23e0aa786-7206513860864277891e0f7159736271 received from {username='slave'} at 127.25.107.5:56199
I20240501 08:11:31.812194 26622 tablet_copy_service.cc:433] P f0f978a65b1549689c460fce4699e7a4: ending tablet copy session 0c0ceab7acf1424aa23a96d23e0aa786-7206513860864277891e0f7159736271 on tablet 7206513860864277891e0f7159736271 with peer 0c0ceab7acf1424aa23a96d23e0aa786
I20240501 08:11:31.841459 27835 raft_consensus.cc:1203] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:11:32.063803 27703 raft_consensus.cc:1203] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 08:11:32.176584 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:11:32.220556 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:11:32.306877 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:11:32.387871 30389 raft_consensus.cc:1050] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587: attempting to promote NON_VOTER f2a0e97dec014a17a4c96bfd2f60a96b to VOTER
I20240501 08:11:32.388831 30389 consensus_queue.cc:237] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:32.391645 27835 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 LEARNER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:32.391997 27965 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:32.392527 26600 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:32.392565 30391 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:32.393227 30388 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:32.393612 30355 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:32.397650 30355 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:32.398500 27965 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:32.401856 26600 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:32.402979 26062 catalog_manager.cc:5484] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 7 to 8, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:32.416838 27835 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f2a0e97dec014a17a4c96bfd2f60a96b (127.25.107.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:32.422603 26472 consensus_queue.cc:237] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } }
I20240501 08:11:32.424530 27835 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:32.424935 27965 raft_consensus.cc:1261] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:32.425128 30355 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:32.427578 30391 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:32.431332 30355 consensus_queue.cc:1035] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:32.432304 26058 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3ddc36d3fc146938c366df2c7f090a6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:32.432802 27835 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:32.433552 27965 raft_consensus.cc:2932] T c3ddc36d3fc146938c366df2c7f090a6 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } }
I20240501 08:11:32.437656 26067 catalog_manager.cc:5484] T c3ddc36d3fc146938c366df2c7f090a6 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 8 to 9, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:32.442905 26582 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ddc36d3fc146938c366df2c7f090a6 with delete_type TABLET_DATA_TOMBSTONED (TS f0f978a65b1549689c460fce4699e7a4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44112
I20240501 08:11:32.443643 30400 tablet_replica.cc:330] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4: stopping tablet replica
I20240501 08:11:32.443871 30400 raft_consensus.cc:2229] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:32.444161 30400 raft_consensus.cc:2258] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:32.445140 30400 ts_tablet_manager.cc:1875] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:32.450377 30400 ts_tablet_manager.cc:1888] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:11:32.450554 30400 log.cc:1196] T c3ddc36d3fc146938c366df2c7f090a6 P f0f978a65b1549689c460fce4699e7a4: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/c3ddc36d3fc146938c366df2c7f090a6
I20240501 08:11:32.451488 26055 catalog_manager.cc:4830] TS f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061): tablet c3ddc36d3fc146938c366df2c7f090a6 (table rebalance_test_table_3 [id=ba459801ecf04f12a4cd0fc429540072]) successfully deleted
W20240501 08:11:32.465600 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:11:32.481942 26472 consensus_queue.cc:237] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:32.484624 27965 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:11:32.484685 26408 consensus_peers.cc:485] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8:33359): Couldn't send request to peer 25e18e28f3914e45be7093bf5bf436e5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e97904ef0a5e41cab9054cbc84aae923. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:32.484717 26600 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:11:32.485198 30388 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:32.485523 30355 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:11:32.488356 30391 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } } }
I20240501 08:11:32.489101 26600 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } } }
I20240501 08:11:32.489081 27965 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } } }
I20240501 08:11:32.494318 26070 catalog_manager.cc:5484] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 6 to 7, NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:32.532667 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:11:32.600827 30430 raft_consensus.cc:1050] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4: attempting to promote NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 to VOTER
I20240501 08:11:32.601774 30430 consensus_queue.cc:237] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:32.604720 27703 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 LEARNER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:32.604786 26472 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:32.605196 27835 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:32.605355 30433 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:32.605782 30431 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:32.606148 30430 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:32.609495 30431 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:32.610515 26472 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:32.610633 27703 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:32.616057 26062 catalog_manager.cc:5484] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786 reported cstate change: config changed from index 7 to 8, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:32.617897 27835 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:32.621137 26600 consensus_queue.cc:237] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:32.623613 27835 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:32.624212 27703 raft_consensus.cc:1261] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 FOLLOWER]: Refusing update from remote peer f0f978a65b1549689c460fce4699e7a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:32.624224 30431 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:32.624732 30430 consensus_queue.cc:1035] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:32.626618 30431 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:32.627279 27835 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P f2a0e97dec014a17a4c96bfd2f60a96b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:32.627681 27703 raft_consensus.cc:2932] T 7206513860864277891e0f7159736271 P 0c0ceab7acf1424aa23a96d23e0aa786 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:32.629794 26055 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7206513860864277891e0f7159736271 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:32.632697 26069 catalog_manager.cc:5484] T 7206513860864277891e0f7159736271 P f0f978a65b1549689c460fce4699e7a4 reported cstate change: config changed from index 8 to 9, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "f0f978a65b1549689c460fce4699e7a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:32.636592 26450 tablet_service.cc:1514] Processing DeleteTablet for tablet 7206513860864277891e0f7159736271 with delete_type TABLET_DATA_TOMBSTONED (TS 875f11971b814f2ea7ce89de1e4f4587 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49828
I20240501 08:11:32.637826 30438 tablet_replica.cc:330] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587: stopping tablet replica
I20240501 08:11:32.638059 30438 raft_consensus.cc:2229] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:32.638553 30438 raft_consensus.cc:2258] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:32.639487 30438 ts_tablet_manager.cc:1875] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:32.644438 30438 ts_tablet_manager.cc:1888] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:11:32.644593 30438 log.cc:1196] T 7206513860864277891e0f7159736271 P 875f11971b814f2ea7ce89de1e4f4587: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/7206513860864277891e0f7159736271
I20240501 08:11:32.645414 26058 catalog_manager.cc:4830] TS 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015): tablet 7206513860864277891e0f7159736271 (table rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]) successfully deleted
I20240501 08:11:32.759183 27965 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:32.763252 26341 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 08:11:32.763415 27835 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 08:11:32.764025 29528 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:11:32.764533 30456 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:11:32.768702 29528 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:32.769694 27835 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
W20240501 08:11:32.771147 27899 consensus_peers.cc:485] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 -> Peer 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5:43333): Couldn't send request to peer 0c0ceab7acf1424aa23a96d23e0aa786. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12641e2d23ac450f9f04e41bb583b32d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:32.774057 26341 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:32.774907 26062 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 reported cstate change: config changed from index 11 to 12, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New cstate: current_term: 2 leader_uuid: "fba0b24a97a84f6fa331664028231d19" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:32.781682 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:11:33.022783 30482 ts_tablet_manager.cc:902] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: Initiating tablet copy from peer 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:11:33.024991 30482 tablet_copy_client.cc:308] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:41015
I20240501 08:11:33.033001 26492 tablet_copy_service.cc:140] P 875f11971b814f2ea7ce89de1e4f4587: Received BeginTabletCopySession request for tablet e97904ef0a5e41cab9054cbc84aae923 from peer 25e18e28f3914e45be7093bf5bf436e5 ({username='slave'} at 127.25.107.8:37843)
I20240501 08:11:33.033210 26492 tablet_copy_service.cc:161] P 875f11971b814f2ea7ce89de1e4f4587: Beginning new tablet copy session on tablet e97904ef0a5e41cab9054cbc84aae923 from peer 25e18e28f3914e45be7093bf5bf436e5 at {username='slave'} at 127.25.107.8:37843: session id = 25e18e28f3914e45be7093bf5bf436e5-e97904ef0a5e41cab9054cbc84aae923
I20240501 08:11:33.035287 26492 tablet_copy_source_session.cc:205] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:33.036648 30482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97904ef0a5e41cab9054cbc84aae923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.041417 30482 tablet_copy_client.cc:783] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:33.041710 30482 tablet_copy_client.cc:647] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:33.043709 30486 tablet_copy_client.cc:1002] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Injecting latency into file download: 1500
I20240501 08:11:33.257067 30499 ts_tablet_manager.cc:902] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: Initiating tablet copy from peer fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815)
I20240501 08:11:33.258396 30499 tablet_copy_client.cc:308] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.7:36815
I20240501 08:11:33.264760 27985 tablet_copy_service.cc:140] P fba0b24a97a84f6fa331664028231d19: Received BeginTabletCopySession request for tablet 12641e2d23ac450f9f04e41bb583b32d from peer 0c0ceab7acf1424aa23a96d23e0aa786 ({username='slave'} at 127.25.107.5:58825)
I20240501 08:11:33.265046 27985 tablet_copy_service.cc:161] P fba0b24a97a84f6fa331664028231d19: Beginning new tablet copy session on tablet 12641e2d23ac450f9f04e41bb583b32d from peer 0c0ceab7acf1424aa23a96d23e0aa786 at {username='slave'} at 127.25.107.5:58825: session id = 0c0ceab7acf1424aa23a96d23e0aa786-12641e2d23ac450f9f04e41bb583b32d
I20240501 08:11:33.267774 27985 tablet_copy_source_session.cc:205] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:33.269522 30499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12641e2d23ac450f9f04e41bb583b32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.274683 30499 tablet_copy_client.cc:783] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:33.274955 30499 tablet_copy_client.cc:647] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:33.276721 30503 tablet_copy_client.cc:1002] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Injecting latency into file download: 1500
W20240501 08:11:33.430832 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:11:34.544126 30482 tablet_copy_client.cc:523] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:34.547032 30482 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap starting.
I20240501 08:11:34.584687 30482 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:34.585132 30482 tablet_bootstrap.cc:492] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: Bootstrap complete.
I20240501 08:11:34.585394 30482 ts_tablet_manager.cc:1367] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: Time spent bootstrapping tablet: real 0.039s user 0.024s sys 0.011s
I20240501 08:11:34.586139 30482 raft_consensus.cc:348] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:34.586395 30482 raft_consensus.cc:729] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 25e18e28f3914e45be7093bf5bf436e5, State: Initialized, Role: LEARNER
I20240501 08:11:34.586603 30482 consensus_queue.cc:260] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: NON_VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: true } }
I20240501 08:11:34.587333 30482 ts_tablet_manager.cc:1398] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:11:34.588106 26492 tablet_copy_service.cc:342] P 875f11971b814f2ea7ce89de1e4f4587: Request end of tablet copy session 25e18e28f3914e45be7093bf5bf436e5-e97904ef0a5e41cab9054cbc84aae923 received from {username='slave'} at 127.25.107.8:37843
I20240501 08:11:34.588259 26492 tablet_copy_service.cc:433] P 875f11971b814f2ea7ce89de1e4f4587: ending tablet copy session 25e18e28f3914e45be7093bf5bf436e5-e97904ef0a5e41cab9054cbc84aae923 on tablet e97904ef0a5e41cab9054cbc84aae923 with peer 25e18e28f3914e45be7093bf5bf436e5
W20240501 08:11:34.667827 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:11:34.671991 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:11:34.722450 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:11:34.751899 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 12641e2d23ac450f9f04e41bb583b32d: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
I20240501 08:11:34.777185 30499 tablet_copy_client.cc:523] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:34.780571 30499 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap starting.
I20240501 08:11:34.823046 30499 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:34.823464 30499 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap complete.
I20240501 08:11:34.823720 30499 ts_tablet_manager.cc:1367] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: Time spent bootstrapping tablet: real 0.043s user 0.040s sys 0.003s
I20240501 08:11:34.824390 30499 raft_consensus.cc:348] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:34.824613 30499 raft_consensus.cc:729] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0c0ceab7acf1424aa23a96d23e0aa786, State: Initialized, Role: LEARNER
I20240501 08:11:34.824788 30499 consensus_queue.cc:260] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:34.825663 30499 ts_tablet_manager.cc:1398] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:34.826710 27985 tablet_copy_service.cc:342] P fba0b24a97a84f6fa331664028231d19: Request end of tablet copy session 0c0ceab7acf1424aa23a96d23e0aa786-12641e2d23ac450f9f04e41bb583b32d received from {username='slave'} at 127.25.107.5:58825
I20240501 08:11:34.826853 27985 tablet_copy_service.cc:433] P fba0b24a97a84f6fa331664028231d19: ending tablet copy session 0c0ceab7acf1424aa23a96d23e0aa786-12641e2d23ac450f9f04e41bb583b32d on tablet 12641e2d23ac450f9f04e41bb583b32d with peer 0c0ceab7acf1424aa23a96d23e0aa786
W20240501 08:11:34.903084 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:11:34.906793 28094 raft_consensus.cc:1203] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 08:11:34.932258 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:11:34.992313 27703 raft_consensus.cc:1203] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
W20240501 08:11:35.231652 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:11:35.292331 30611 raft_consensus.cc:1050] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587: attempting to promote NON_VOTER 25e18e28f3914e45be7093bf5bf436e5 to VOTER
I20240501 08:11:35.293109 30611 consensus_queue.cc:237] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } }
I20240501 08:11:35.295919 27965 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:35.296001 28094 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5 [term 1 LEARNER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:35.296379 26600 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:11:35.296609 30610 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:35.296983 30613 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:35.297292 30611 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:11:35.305902 30613 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:35.306777 27965 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:35.306869 26600 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:35.311255 26062 catalog_manager.cc:5484] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 reported cstate change: config changed from index 7 to 8, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:35.313186 28094 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:35.318755 26472 consensus_queue.cc:237] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } }
I20240501 08:11:35.320861 28094 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:35.321050 27965 raft_consensus.cc:1261] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:11:35.321693 30611 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:35.322006 30391 consensus_queue.cc:1035] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:11:35.324424 30391 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:35.325498 27965 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P fba0b24a97a84f6fa331664028231d19 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:35.325708 28094 raft_consensus.cc:2932] T e97904ef0a5e41cab9054cbc84aae923 P 25e18e28f3914e45be7093bf5bf436e5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } }
I20240501 08:11:35.328236 26058 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e97904ef0a5e41cab9054cbc84aae923 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:35.330631 26070 catalog_manager.cc:5484] T e97904ef0a5e41cab9054cbc84aae923 P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 8 to 9, VOTER f0f978a65b1549689c460fce4699e7a4 (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:35.334350 26582 tablet_service.cc:1514] Processing DeleteTablet for tablet e97904ef0a5e41cab9054cbc84aae923 with delete_type TABLET_DATA_TOMBSTONED (TS f0f978a65b1549689c460fce4699e7a4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44112
I20240501 08:11:35.335139 30621 tablet_replica.cc:330] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4: stopping tablet replica
I20240501 08:11:35.335336 30621 raft_consensus.cc:2229] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:35.335712 30621 raft_consensus.cc:2258] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:35.336613 30621 ts_tablet_manager.cc:1875] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:35.341614 30621 ts_tablet_manager.cc:1888] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:11:35.341730 30621 log.cc:1196] T e97904ef0a5e41cab9054cbc84aae923 P f0f978a65b1549689c460fce4699e7a4: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/e97904ef0a5e41cab9054cbc84aae923
I20240501 08:11:35.342681 26055 catalog_manager.cc:4830] TS f0f978a65b1549689c460fce4699e7a4 (127.25.107.4:37061): tablet e97904ef0a5e41cab9054cbc84aae923 (table rebalance_test_table_1 [id=c0c368690d524dc99ba6d028425bc8dc]) successfully deleted
I20240501 08:11:35.595423 30638 raft_consensus.cc:1050] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19: attempting to promote NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 to VOTER
I20240501 08:11:35.596473 30638 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:35.599465 27835 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 08:11:35.599638 26341 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 08:11:35.599960 27703 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 LEARNER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 08:11:35.600085 30637 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:11:35.600492 30638 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:11:35.600836 30563 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:11:35.604022 30638 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:35.604766 26341 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:35.608162 27835 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:35.609333 26070 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 reported cstate change: config changed from index 12 to 13, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fba0b24a97a84f6fa331664028231d19" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:35.612761 27703 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } attrs { replace: true } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:35.617602 27965 consensus_queue.cc:237] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:35.619757 27835 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 08:11:35.619762 27703 raft_consensus.cc:1261] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 FOLLOWER]: Refusing update from remote peer fba0b24a97a84f6fa331664028231d19: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 08:11:35.620266 30563 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:11:35.620601 30637 consensus_queue.cc:1035] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:11:35.622822 30563 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:35.623450 27702 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P 0c0ceab7acf1424aa23a96d23e0aa786 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:35.623720 27835 raft_consensus.cc:2932] T 12641e2d23ac450f9f04e41bb583b32d P f2a0e97dec014a17a4c96bfd2f60a96b [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:35.626483 26058 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12641e2d23ac450f9f04e41bb583b32d with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:35.628480 26069 catalog_manager.cc:5484] T 12641e2d23ac450f9f04e41bb583b32d P fba0b24a97a84f6fa331664028231d19 reported cstate change: config changed from index 13 to 14, VOTER bc13665cb8894b1abf18738f527ab244 (127.25.107.2) evicted. New cstate: current_term: 2 leader_uuid: "fba0b24a97a84f6fa331664028231d19" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a0e97dec014a17a4c96bfd2f60a96b" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 34341 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:35.632063 26322 tablet_service.cc:1514] Processing DeleteTablet for tablet 12641e2d23ac450f9f04e41bb583b32d with delete_type TABLET_DATA_TOMBSTONED (TS bc13665cb8894b1abf18738f527ab244 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:33068
I20240501 08:11:35.632835 30646 tablet_replica.cc:330] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244: stopping tablet replica
I20240501 08:11:35.633014 30646 raft_consensus.cc:2229] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:35.633358 30646 raft_consensus.cc:2258] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:35.634150 30646 ts_tablet_manager.cc:1875] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:35.638592 30646 ts_tablet_manager.cc:1888] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240501 08:11:35.638736 30646 log.cc:1196] T 12641e2d23ac450f9f04e41bb583b32d P bc13665cb8894b1abf18738f527ab244: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/12641e2d23ac450f9f04e41bb583b32d
I20240501 08:11:35.639534 26055 catalog_manager.cc:4830] TS bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319): tablet 12641e2d23ac450f9f04e41bb583b32d (table rebalance_test_table_2 [id=d8e7ec50bf7d41198b047401a7389516]) successfully deleted
I20240501 08:11:35.705581 26472 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:35.708115 27965 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 3 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 08:11:35.708173 28094 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 3 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 08:11:35.708734 30611 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:11:35.709069 30391 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:11:35.713800 30391 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
W20240501 08:11:35.714752 26406 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 -> Peer 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5:43333): Couldn't send request to peer 0c0ceab7acf1424aa23a96d23e0aa786. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 359983c3db9a43be92e135c4b722dc0f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:35.715195 28094 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:35.715513 27965 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } } }
I20240501 08:11:35.720335 26062 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 reported cstate change: config changed from index 12 to 13, NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) added. New cstate: current_term: 3 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:35.820065 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:11:36.202765 30678 ts_tablet_manager.cc:902] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: Initiating tablet copy from peer 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015)
I20240501 08:11:36.204279 30678 tablet_copy_client.cc:308] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:41015
I20240501 08:11:36.210392 26492 tablet_copy_service.cc:140] P 875f11971b814f2ea7ce89de1e4f4587: Received BeginTabletCopySession request for tablet 359983c3db9a43be92e135c4b722dc0f from peer 0c0ceab7acf1424aa23a96d23e0aa786 ({username='slave'} at 127.25.107.5:43195)
I20240501 08:11:36.210579 26492 tablet_copy_service.cc:161] P 875f11971b814f2ea7ce89de1e4f4587: Beginning new tablet copy session on tablet 359983c3db9a43be92e135c4b722dc0f from peer 0c0ceab7acf1424aa23a96d23e0aa786 at {username='slave'} at 127.25.107.5:43195: session id = 0c0ceab7acf1424aa23a96d23e0aa786-359983c3db9a43be92e135c4b722dc0f
I20240501 08:11:36.212569 26492 tablet_copy_source_session.cc:205] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:36.214058 30678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359983c3db9a43be92e135c4b722dc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:36.218986 30678 tablet_copy_client.cc:783] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:36.219250 30678 tablet_copy_client.cc:647] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:36.220760 30682 tablet_copy_client.cc:1002] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Injecting latency into file download: 1500
W20240501 08:11:36.260046 26538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 08:11:36.617660 26408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 08:11:36.829342 26278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:11:37.010778 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:11:37.063058 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:11:37.184348 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:11:37.493508 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:11:37.546798 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:11:37.721252 30678 tablet_copy_client.cc:523] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:37.724211 30678 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap starting.
W20240501 08:11:37.760628 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:11:37.761036 30678 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:37.761394 30678 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: Bootstrap complete.
I20240501 08:11:37.761643 30678 ts_tablet_manager.cc:1367] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.009s
I20240501 08:11:37.762313 30678 raft_consensus.cc:348] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:37.762534 30678 raft_consensus.cc:729] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 0c0ceab7acf1424aa23a96d23e0aa786, State: Initialized, Role: LEARNER
I20240501 08:11:37.762725 30678 consensus_queue.cc:260] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: NON_VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: true } }
I20240501 08:11:37.763471 30678 ts_tablet_manager.cc:1398] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 08:11:37.764513 26492 tablet_copy_service.cc:342] P 875f11971b814f2ea7ce89de1e4f4587: Request end of tablet copy session 0c0ceab7acf1424aa23a96d23e0aa786-359983c3db9a43be92e135c4b722dc0f received from {username='slave'} at 127.25.107.5:43195
I20240501 08:11:37.764673 26492 tablet_copy_service.cc:433] P 875f11971b814f2ea7ce89de1e4f4587: ending tablet copy session 0c0ceab7acf1424aa23a96d23e0aa786-359983c3db9a43be92e135c4b722dc0f on tablet 359983c3db9a43be92e135c4b722dc0f with peer 0c0ceab7acf1424aa23a96d23e0aa786
I20240501 08:11:37.806528 27702 raft_consensus.cc:1203] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13] Dedup: 3.13->[]
W20240501 08:11:38.045169 26058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:11:38.046525 26058 catalog_manager.cc:4573] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9981240ef4c1466cb37895e24a0c6e5c: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111)
W20240501 08:11:38.284615 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:11:38.328342 30731 raft_consensus.cc:1050] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: attempting to promote NON_VOTER 0c0ceab7acf1424aa23a96d23e0aa786 to VOTER
I20240501 08:11:38.329020 30731 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:38.332036 27702 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [term 3 LEARNER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 08:11:38.332099 28094 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 3 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 08:11:38.332623 27965 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 3 FOLLOWER]: Refusing update from remote peer 875f11971b814f2ea7ce89de1e4f4587: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 08:11:38.332687 30731 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:11:38.333364 30753 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:11:38.333791 30751 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:11:38.336393 30753 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:38.337020 27702 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:38.337844 27965 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:38.341818 28094 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:38.342561 26069 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 reported cstate change: config changed from index 13 to 14, 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "875f11971b814f2ea7ce89de1e4f4587" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:38.475438 26472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f"
dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53004
I20240501 08:11:38.475682 26472 raft_consensus.cc:595] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 LEADER]: Received request to transfer leadership
I20240501 08:11:38.694036 26472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f"
dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53030
I20240501 08:11:38.694307 26472 raft_consensus.cc:595] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 LEADER]: Received request to transfer leadership
I20240501 08:11:38.779104 30751 raft_consensus.cc:979] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: : Instructing follower 25e18e28f3914e45be7093bf5bf436e5 to start an election
I20240501 08:11:38.779307 30751 raft_consensus.cc:1067] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 LEADER]: Signalling peer 25e18e28f3914e45be7093bf5bf436e5 to start an election
I20240501 08:11:38.780092 28094 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f"
dest_uuid: "25e18e28f3914e45be7093bf5bf436e5"
from {username='slave'} at 127.25.107.3:47059
I20240501 08:11:38.780372 28094 raft_consensus.cc:482] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:38.780516 28094 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:11:38.782529 28094 raft_consensus.cc:504] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:38.784171 28094 leader_election.cc:288] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [CANDIDATE]: Term 4 election: Requested vote from peers 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015), fba0b24a97a84f6fa331664028231d19 (127.25.107.7:36815), 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5:43333)
I20240501 08:11:38.784694 26472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "25e18e28f3914e45be7093bf5bf436e5" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "875f11971b814f2ea7ce89de1e4f4587"
I20240501 08:11:38.784942 26472 raft_consensus.cc:3032] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:11:38.785027 26472 raft_consensus.cc:729] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 875f11971b814f2ea7ce89de1e4f4587, State: Running, Role: LEADER
I20240501 08:11:38.785382 26472 consensus_queue.cc:260] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:38.785904 26472 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:11:38.789187 26472 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e18e28f3914e45be7093bf5bf436e5 in term 4.
I20240501 08:11:38.793833 27702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "25e18e28f3914e45be7093bf5bf436e5" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "0c0ceab7acf1424aa23a96d23e0aa786"
I20240501 08:11:38.794157 27702 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:11:38.794150 27965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "359983c3db9a43be92e135c4b722dc0f" candidate_uuid: "25e18e28f3914e45be7093bf5bf436e5" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "fba0b24a97a84f6fa331664028231d19"
I20240501 08:11:38.794394 27965 raft_consensus.cc:3037] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:11:38.797111 27965 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e18e28f3914e45be7093bf5bf436e5 in term 4.
I20240501 08:11:38.797165 27702 raft_consensus.cc:2454] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e18e28f3914e45be7093bf5bf436e5 in term 4.
I20240501 08:11:38.797796 28029 leader_election.cc:302] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 4 yes votes; 0 no votes. yes voters: 0c0ceab7acf1424aa23a96d23e0aa786, 25e18e28f3914e45be7093bf5bf436e5, 875f11971b814f2ea7ce89de1e4f4587, fba0b24a97a84f6fa331664028231d19; no voters:
I20240501 08:11:38.798313 30759 raft_consensus.cc:2781] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:11:38.798915 30759 raft_consensus.cc:686] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 4 LEADER]: Becoming Leader. State: Replica: 25e18e28f3914e45be7093bf5bf436e5, State: Running, Role: LEADER
I20240501 08:11:38.799492 30759 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:38.802807 26067 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 reported cstate change: term changed from 3 to 4, leader changed from 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) to 25e18e28f3914e45be7093bf5bf436e5 (127.25.107.8). New cstate: current_term: 4 leader_uuid: "25e18e28f3914e45be7093bf5bf436e5" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:39.187793 26472 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 4 FOLLOWER]: Refusing update from remote peer 25e18e28f3914e45be7093bf5bf436e5: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240501 08:11:39.188424 30759 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 08:11:39.221908 27965 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 4 FOLLOWER]: Refusing update from remote peer 25e18e28f3914e45be7093bf5bf436e5: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240501 08:11:39.222548 30759 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 08:11:39.226521 27702 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [term 4 FOLLOWER]: Refusing update from remote peer 25e18e28f3914e45be7093bf5bf436e5: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240501 08:11:39.227332 30788 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240501 08:11:39.318732 26408 consensus_peers.cc:485] T 697a56042bc14439b2ed2fa559e06a01 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:11:39.644061 26408 consensus_peers.cc:485] T 0b1a24c1d65242e9a634c7f1f99fff02 P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:11:39.699177 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:11:39.726962 28094 consensus_queue.cc:237] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 4.15, Last appended by leader: 14, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } }
I20240501 08:11:39.729001 27965 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 4 FOLLOWER]: Refusing update from remote peer 25e18e28f3914e45be7093bf5bf436e5: Log matching property violated. Preceding OpId in replica: term: 4 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I20240501 08:11:39.729014 27702 raft_consensus.cc:1261] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [term 4 FOLLOWER]: Refusing update from remote peer 25e18e28f3914e45be7093bf5bf436e5: Log matching property violated. Preceding OpId in replica: term: 4 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I20240501 08:11:39.729590 30802 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 08:11:39.729924 30788 consensus_queue.cc:1035] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 08:11:39.732093 30802 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 [term 4 LEADER]: Committing config change with OpId 4.16: config changed from index 14 to 16, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:39.732720 27965 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P fba0b24a97a84f6fa331664028231d19 [term 4 FOLLOWER]: Committing config change with OpId 4.16: config changed from index 14 to 16, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:39.733258 27702 raft_consensus.cc:2932] T 359983c3db9a43be92e135c4b722dc0f P 0c0ceab7acf1424aa23a96d23e0aa786 [term 4 FOLLOWER]: Committing config change with OpId 4.16: config changed from index 14 to 16, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } } }
I20240501 08:11:39.736030 26058 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 359983c3db9a43be92e135c4b722dc0f with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:11:39.737670 26062 catalog_manager.cc:5484] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 reported cstate change: config changed from index 14 to 16, VOTER 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3) evicted. New cstate: current_term: 4 leader_uuid: "25e18e28f3914e45be7093bf5bf436e5" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "25e18e28f3914e45be7093bf5bf436e5" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 33359 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c0ceab7acf1424aa23a96d23e0aa786" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 43333 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:39.742838 26450 tablet_service.cc:1514] Processing DeleteTablet for tablet 359983c3db9a43be92e135c4b722dc0f with delete_type TABLET_DATA_TOMBSTONED (TS 875f11971b814f2ea7ce89de1e4f4587 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:49828
I20240501 08:11:39.743611 30828 tablet_replica.cc:330] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: stopping tablet replica
I20240501 08:11:39.743889 30828 raft_consensus.cc:2229] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:39.744191 30828 raft_consensus.cc:2258] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:39.745061 30828 ts_tablet_manager.cc:1875] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:39.750392 30828 ts_tablet_manager.cc:1888] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.15
I20240501 08:11:39.750607 30828 log.cc:1196] T 359983c3db9a43be92e135c4b722dc0f P 875f11971b814f2ea7ce89de1e4f4587: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/359983c3db9a43be92e135c4b722dc0f
I20240501 08:11:39.751415 26058 catalog_manager.cc:4830] TS 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015): tablet 359983c3db9a43be92e135c4b722dc0f (table rebalance_test_table_4 [id=c5ed59f460e0476e8fc6e07548fb561c]) successfully deleted
I20240501 08:11:39.857250 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:39.871544 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:39.882357 28075 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:39.902693 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:11:39.908533 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:11:39.909374 27945 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:11:39.917269 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:11:39.927775 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:39.955240 26322 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:40.116369 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:40.134302 26322 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:40.136700 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:40.144850 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:40.149948 28075 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:40.152796 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:40.198769 27945 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:11:40.378159 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:11:40.448163 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.1:40425
--local_ip_for_outbound_sockets=127.25.107.1
--tserver_master_addrs=127.25.107.62:37731
--webserver_port=37989
--webserver_interface=127.25.107.1
--builtin_ntp_servers=127.25.107.20:41669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 08:11:40.791946 30905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:40.792287 30905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:40.792449 30905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:40.800467 30905 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:40.800627 30905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:40.800789 30905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.1
I20240501 08:11:40.810330 30905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.1:40425
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.107.1
--webserver_port=37989
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.107.62:37731
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.1
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:40.811756 30905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:40.813566 30905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:40.826373 30914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:40.826414 30912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:40.826565 30911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:40.826956 30905 server_base.cc:1034] running on GCE node
I20240501 08:11:40.827735 30905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:40.828722 30905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:40.829936 30905 hybrid_clock.cc:648] HybridClock initialized: now 1714551100829868 us; error 57 us; skew 500 ppm
I20240501 08:11:40.830338 30905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:40.836908 30905 webserver.cc:460] Webserver started at http://127.25.107.1:37989/ using document root <none> and password file <none>
I20240501 08:11:40.837757 30905 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:40.837895 30905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:40.843811 30905 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 08:11:40.848309 30920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:40.850039 30905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:40.850281 30905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c0624af4925d4686b5896f487f864943"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-dv41"
I20240501 08:11:40.851372 30905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240501 08:11:40.867204 26538 consensus_peers.cc:485] T d6fff31fdd6b496198695328538e11f0 P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:11:40.885712 30905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:40.886627 30905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:40.886965 30905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:40.888074 30905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:40.890285 30927 ts_tablet_manager.cc:522] Loading tablet metadata (0/12 complete)
I20240501 08:11:40.918136 30905 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240501 08:11:40.918278 30905 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.028s user 0.001s sys 0.000s
I20240501 08:11:40.918414 30905 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240501 08:11:40.921823 30927 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:40.947469 30927 log.cc:824] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:40.948019 30905 ts_tablet_manager.cc:590] Registered 12 tablets
I20240501 08:11:40.948149 30905 ts_tablet_manager.cc:569] Time spent register tablets: real 0.030s user 0.026s sys 0.002s
I20240501 08:11:41.003793 30927 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.004592 30927 tablet_bootstrap.cc:492] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.006160 30927 ts_tablet_manager.cc:1367] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.085s user 0.077s sys 0.004s
I20240501 08:11:41.013628 30927 raft_consensus.cc:348] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:11:41.014668 30927 raft_consensus.cc:729] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.015825 30927 consensus_queue.cc:260] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:11:41.018816 30927 ts_tablet_manager.cc:1398] T 9981240ef4c1466cb37895e24a0c6e5c P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20240501 08:11:41.019333 30927 tablet_bootstrap.cc:492] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.033546 30905 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.1:40425
I20240501 08:11:41.033596 31037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.1:40425 every 8 connection(s)
I20240501 08:11:41.035221 30905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:11:41.038152 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 30905
I20240501 08:11:41.060338 31039 heartbeater.cc:347] Connected to a master server at 127.25.107.62:37731
I20240501 08:11:41.060683 31039 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:41.061597 31039 heartbeater.cc:511] Master 127.25.107.62:37731 requested a full tablet report, sending...
I20240501 08:11:41.064554 26067 ts_manager.cc:194] Re-registered known tserver with Master: c0624af4925d4686b5896f487f864943 (127.25.107.1:40425)
I20240501 08:11:41.067726 26067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.1:50219
I20240501 08:11:41.075548 30927 tablet_bootstrap.cc:492] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.076146 30927 tablet_bootstrap.cc:492] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.077329 30927 ts_tablet_manager.cc:1367] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.058s user 0.046s sys 0.008s
I20240501 08:11:41.078404 30927 raft_consensus.cc:348] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:41.078629 30927 raft_consensus.cc:729] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.078896 30927 consensus_queue.cc:260] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:41.079377 30927 ts_tablet_manager.cc:1398] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:11:41.079424 31039 heartbeater.cc:503] Master 127.25.107.62:37731 was elected leader, sending a full tablet report...
I20240501 08:11:41.079671 30927 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.132673 30927 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.133275 30927 tablet_bootstrap.cc:492] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.134519 30927 ts_tablet_manager.cc:1367] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.055s user 0.038s sys 0.016s
I20240501 08:11:41.135371 30927 raft_consensus.cc:348] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:11:41.135648 30927 raft_consensus.cc:729] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.135934 30927 consensus_queue.cc:260] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:11:41.136477 30927 ts_tablet_manager.cc:1398] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:41.137014 30927 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.178428 30927 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.178896 30927 tablet_bootstrap.cc:492] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.179749 30927 ts_tablet_manager.cc:1367] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.043s user 0.033s sys 0.008s
I20240501 08:11:41.180420 30927 raft_consensus.cc:348] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:41.180668 30927 raft_consensus.cc:729] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.180943 30927 consensus_queue.cc:260] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "fba0b24a97a84f6fa331664028231d19" member_type: NON_VOTER last_known_addr { host: "127.25.107.7" port: 36815 } attrs { promote: true } }
I20240501 08:11:41.181497 30927 ts_tablet_manager.cc:1398] T 12641e2d23ac450f9f04e41bb583b32d P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:41.181802 30927 tablet_bootstrap.cc:492] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.219698 30927 tablet_bootstrap.cc:492] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.220155 30927 tablet_bootstrap.cc:492] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.220994 30927 ts_tablet_manager.cc:1367] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.008s
I20240501 08:11:41.221711 30927 raft_consensus.cc:348] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:41.221902 30927 raft_consensus.cc:729] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.222189 30927 consensus_queue.cc:260] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:41.222690 30927 ts_tablet_manager.cc:1398] T d6fff31fdd6b496198695328538e11f0 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:11:41.222914 30927 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.262033 30927 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.262545 30927 tablet_bootstrap.cc:492] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.263396 30927 ts_tablet_manager.cc:1367] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.041s user 0.031s sys 0.008s
I20240501 08:11:41.264082 30927 raft_consensus.cc:348] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:41.264266 30927 raft_consensus.cc:729] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.264495 30927 consensus_queue.cc:260] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:41.264962 30927 ts_tablet_manager.cc:1398] T 595048c2a092461b979fe7e107882d75 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:41.265323 30927 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.306854 30927 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.307463 30927 tablet_bootstrap.cc:492] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.308650 30927 ts_tablet_manager.cc:1367] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.043s user 0.037s sys 0.004s
I20240501 08:11:41.309676 30970 tablet_service.cc:1514] Processing DeleteTablet for tablet a5b85279a473435fb21f7e6c3cbebea2 with delete_type TABLET_DATA_TOMBSTONED (TS c0624af4925d4686b5896f487f864943 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56628
I20240501 08:11:41.310547 30927 raft_consensus.cc:348] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:11:41.310729 31051 tablet_replica.cc:330] stopping tablet replica
I20240501 08:11:41.310837 30927 raft_consensus.cc:729] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.311173 31051 raft_consensus.cc:2229] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:41.311115 30927 consensus_queue.cc:260] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:11:41.311671 31051 raft_consensus.cc:2258] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:41.311772 30927 ts_tablet_manager.cc:1398] T 359983c3db9a43be92e135c4b722dc0f P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:41.312073 30927 tablet_bootstrap.cc:492] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.313220 31051 ts_tablet_manager.cc:1875] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:41.321727 31051 ts_tablet_manager.cc:1888] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:11:41.321903 31051 log.cc:1196] T a5b85279a473435fb21f7e6c3cbebea2 P c0624af4925d4686b5896f487f864943: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/a5b85279a473435fb21f7e6c3cbebea2
I20240501 08:11:41.322861 26058 catalog_manager.cc:4830] TS c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): tablet a5b85279a473435fb21f7e6c3cbebea2 (table rebalance_test_table_0 [id=297962d4e37d4755a89f11513661380f]) successfully deleted
I20240501 08:11:41.365330 30927 tablet_bootstrap.cc:492] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.365959 30927 tablet_bootstrap.cc:492] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.367162 30927 ts_tablet_manager.cc:1367] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.055s user 0.046s sys 0.008s
I20240501 08:11:41.367990 30927 raft_consensus.cc:348] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:11:41.368217 30927 raft_consensus.cc:729] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.368516 30927 consensus_queue.cc:260] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } }
I20240501 08:11:41.369134 30927 ts_tablet_manager.cc:1398] T c57c173fdd4444d6841410fc1adb7674 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:41.369518 30927 tablet_bootstrap.cc:492] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.417047 30992 raft_consensus.cc:3037] T 697a56042bc14439b2ed2fa559e06a01 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:41.422919 30927 tablet_bootstrap.cc:492] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.424038 30927 tablet_bootstrap.cc:492] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.425647 30927 ts_tablet_manager.cc:1367] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.056s user 0.050s sys 0.004s
I20240501 08:11:41.426542 30927 raft_consensus.cc:348] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:11:41.426787 30927 raft_consensus.cc:729] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.427088 30927 consensus_queue.cc:260] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:11:41.427718 30927 ts_tablet_manager.cc:1398] T 99c45caee3514df08b819cf61954f88d P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:41.427964 30927 tablet_bootstrap.cc:492] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.471571 30927 tablet_bootstrap.cc:492] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.472092 30927 tablet_bootstrap.cc:492] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.472887 30927 ts_tablet_manager.cc:1367] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.045s user 0.034s sys 0.008s
I20240501 08:11:41.473592 30927 raft_consensus.cc:348] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:41.473778 30927 raft_consensus.cc:729] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.474031 30927 consensus_queue.cc:260] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } }
I20240501 08:11:41.474561 30927 ts_tablet_manager.cc:1398] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:11:41.474920 30927 tablet_bootstrap.cc:492] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.522747 30927 tablet_bootstrap.cc:492] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.523330 30927 tablet_bootstrap.cc:492] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.524494 30927 ts_tablet_manager.cc:1367] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.050s user 0.039s sys 0.008s
I20240501 08:11:41.525314 30927 raft_consensus.cc:348] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:11:41.525559 30927 raft_consensus.cc:729] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.525810 30927 consensus_queue.cc:260] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "875f11971b814f2ea7ce89de1e4f4587" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 41015 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } }
I20240501 08:11:41.526495 30927 ts_tablet_manager.cc:1398] T 0b1a24c1d65242e9a634c7f1f99fff02 P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:11:41.526777 30927 tablet_bootstrap.cc:492] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943: Bootstrap starting.
I20240501 08:11:41.572628 30927 tablet_bootstrap.cc:492] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:41.573177 30927 tablet_bootstrap.cc:492] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943: Bootstrap complete.
I20240501 08:11:41.574246 30927 ts_tablet_manager.cc:1367] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943: Time spent bootstrapping tablet: real 0.048s user 0.036s sys 0.008s
I20240501 08:11:41.575071 30927 raft_consensus.cc:348] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:11:41.575306 30927 raft_consensus.cc:729] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0624af4925d4686b5896f487f864943, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.575567 30927 consensus_queue.cc:260] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f978a65b1549689c460fce4699e7a4" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 37061 } } peers { permanent_uuid: "c0624af4925d4686b5896f487f864943" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 40425 } } peers { permanent_uuid: "bc13665cb8894b1abf18738f527ab244" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 34319 } }
I20240501 08:11:41.576179 30927 ts_tablet_manager.cc:1398] T f01a4fd9d73849ac8ac01c7a6470fb2c P c0624af4925d4686b5896f487f864943: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:41.658887 28075 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:41.661890 26322 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:41.663445 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:41.665699 30970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:41.674780 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:41.675472 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:41.684397 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:41.698341 27945 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:41.862421 30992 raft_consensus.cc:3037] T 5bc10e8d77e54d6f959552a962abee60 P c0624af4925d4686b5896f487f864943 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:41.939868 27683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:41.946354 26322 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:41.957870 27815 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:41.958783 28075 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:41.960811 26450 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:41.980953 26582 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:42.005367 30970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:42.011437 27945 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:42.200721 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 30905
I20240501 08:11:42.207803 31033 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:42.322563 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 30905
W20240501 08:11:42.342144 26278 connection.cc:535] client connection to 127.25.107.1:40425 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:11:42.342386 26278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 8 similar messages]
I20240501 08:11:42.342577 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 26260
W20240501 08:11:42.344211 26278 consensus_peers.cc:485] T c57c173fdd4444d6841410fc1adb7674 P bc13665cb8894b1abf18738f527ab244 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:42.348986 26381 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:11:42.403016 26408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240501 08:11:42.405215 26408 consensus_peers.cc:485] T 99c45caee3514df08b819cf61954f88d P 875f11971b814f2ea7ce89de1e4f4587 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:42.539992 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 26260
W20240501 08:11:42.545833 26538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 08:11:42.547071 26538 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer c0624af4925d4686b5896f487f864943 (127.25.107.1:40425): Couldn't send request to peer c0624af4925d4686b5896f487f864943. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.1:40425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:11:42.570003 28029 connection.cc:535] server connection from 127.25.107.2:36695 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:11:42.570003 27900 connection.cc:535] server connection from 127.25.107.2:37041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:11:42.570964 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 26390
I20240501 08:11:42.577397 26512 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:11:42.640138 26535 consensus_peers.cc:485] T f01a4fd9d73849ac8ac01c7a6470fb2c P f0f978a65b1549689c460fce4699e7a4 -> Peer bc13665cb8894b1abf18738f527ab244 (127.25.107.2:34319): Couldn't send request to peer bc13665cb8894b1abf18738f527ab244. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.2:34319: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:42.770043 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 26390
W20240501 08:11:42.800546 27770 connection.cc:535] server connection from 127.25.107.3:39357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:11:42.801569 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 26520
W20240501 08:11:42.802120 26538 consensus_peers.cc:485] T 5bc10e8d77e54d6f959552a962abee60 P f0f978a65b1549689c460fce4699e7a4 -> Peer 875f11971b814f2ea7ce89de1e4f4587 (127.25.107.3:41015): Couldn't send request to peer 875f11971b814f2ea7ce89de1e4f4587. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.3:41015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:42.807565 26641 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:42.995476 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 26520
I20240501 08:11:43.024128 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 27621
I20240501 08:11:43.030087 27743 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:43.159183 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 27621
I20240501 08:11:43.179826 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 27752
I20240501 08:11:43.185935 27874 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:43.312717 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 27752
I20240501 08:11:43.334033 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 27883
I20240501 08:11:43.340360 28004 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:11:43.363447 28029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:43333: connect: Connection refused (error 111)
W20240501 08:11:43.365063 28029 consensus_peers.cc:485] T 359983c3db9a43be92e135c4b722dc0f P 25e18e28f3914e45be7093bf5bf436e5 -> Peer 0c0ceab7acf1424aa23a96d23e0aa786 (127.25.107.5:43333): Couldn't send request to peer 0c0ceab7acf1424aa23a96d23e0aa786. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:43333: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:43.482118 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 27883
I20240501 08:11:43.505129 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 28013
I20240501 08:11:43.511919 28133 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:43.643702 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 28013
I20240501 08:11:43.665015 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 26040
I20240501 08:11:43.666123 26101 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:43.838573 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 26040
2024-05-01T08:11:43Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (64367 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (64368 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240501 08:11:43.905469 26028 test_util.cc:268] Using random seed: 159342230
I20240501 08:11:43.907274 26028 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:11:43.907368 26028 ts_itest-base.cc:113] --------------
I20240501 08:11:43.907405 26028 ts_itest-base.cc:114] 5 tablet servers
I20240501 08:11:43.907475 26028 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:11:43.907522 26028 ts_itest-base.cc:116] --------------
2024-05-01T08:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:11:43Z Disabled control of system clock
I20240501 08:11:43.933183 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.107.62:39107
--webserver_interface=127.25.107.62
--webserver_port=0
--builtin_ntp_servers=127.25.107.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.107.62:39107
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 08:11:44.267477 31192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:44.267799 31192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:44.267939 31192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:44.275565 31192 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:44.275712 31192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:11:44.275780 31192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:44.275858 31192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:11:44.275914 31192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:11:44.285312 31192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.25.107.62:39107
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.107.62:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.107.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:44.286767 31192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:44.288578 31192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:44.301781 31200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:44.301774 31198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:44.301818 31197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:44.302702 31192 server_base.cc:1034] running on GCE node
I20240501 08:11:44.303519 31192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:44.305038 31192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:44.306308 31192 hybrid_clock.cc:648] HybridClock initialized: now 1714551104306224 us; error 44 us; skew 500 ppm
I20240501 08:11:44.306689 31192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:44.309094 31192 webserver.cc:460] Webserver started at http://127.25.107.62:40193/ using document root <none> and password file <none>
I20240501 08:11:44.309878 31192 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:44.309991 31192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:44.310361 31192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:44.313117 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "382ed04d394749f38b3b21e7602e7e8c"
format_stamp: "Formatted at 2024-05-01 08:11:44 on dist-test-slave-dv41"
I20240501 08:11:44.313957 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "382ed04d394749f38b3b21e7602e7e8c"
format_stamp: "Formatted at 2024-05-01 08:11:44 on dist-test-slave-dv41"
I20240501 08:11:44.319617 31192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 08:11:44.323267 31206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:44.324765 31192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:11:44.324955 31192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
uuid: "382ed04d394749f38b3b21e7602e7e8c"
format_stamp: "Formatted at 2024-05-01 08:11:44 on dist-test-slave-dv41"
I20240501 08:11:44.325171 31192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:44.341656 31192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:44.342667 31192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:44.342952 31192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:44.360332 31192 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.62:39107
I20240501 08:11:44.360357 31257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.62:39107 every 8 connection(s)
I20240501 08:11:44.361853 31192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:11:44.365824 31258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.371062 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 31192
I20240501 08:11:44.371312 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:11:44.376865 31258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c: Bootstrap starting.
I20240501 08:11:44.380502 31258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.381915 31258 log.cc:824] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:44.385279 31258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c: No bootstrap required, opened a new log
I20240501 08:11:44.391773 31258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382ed04d394749f38b3b21e7602e7e8c" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 39107 } }
I20240501 08:11:44.392210 31258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.392321 31258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382ed04d394749f38b3b21e7602e7e8c, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.392980 31258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382ed04d394749f38b3b21e7602e7e8c" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 39107 } }
I20240501 08:11:44.393205 31258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:44.393330 31258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:44.393476 31258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.395594 31258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382ed04d394749f38b3b21e7602e7e8c" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 39107 } }
I20240501 08:11:44.396112 31258 leader_election.cc:302] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382ed04d394749f38b3b21e7602e7e8c; no voters:
I20240501 08:11:44.396764 31258 leader_election.cc:288] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:44.396908 31263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:44.397552 31263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [term 1 LEADER]: Becoming Leader. State: Replica: 382ed04d394749f38b3b21e7602e7e8c, State: Running, Role: LEADER
I20240501 08:11:44.398119 31263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382ed04d394749f38b3b21e7602e7e8c" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 39107 } }
I20240501 08:11:44.398895 31258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:44.400887 31264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "382ed04d394749f38b3b21e7602e7e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382ed04d394749f38b3b21e7602e7e8c" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 39107 } } }
I20240501 08:11:44.401290 31264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:44.401795 31265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382ed04d394749f38b3b21e7602e7e8c. Latest consensus state: current_term: 1 leader_uuid: "382ed04d394749f38b3b21e7602e7e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382ed04d394749f38b3b21e7602e7e8c" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 39107 } } }
I20240501 08:11:44.403687 31265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:44.405143 31269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:44.410001 31269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:44.419827 31269 catalog_manager.cc:1347] Generated new cluster ID: 9adb0576b7684256b0a603e160219c41
I20240501 08:11:44.420022 31269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:44.434913 31269 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:44.436192 31269 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:44.452399 31269 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 382ed04d394749f38b3b21e7602e7e8c: Generated new TSK 0
I20240501 08:11:44.453395 31269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:44.469972 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.1:0
--local_ip_for_outbound_sockets=127.25.107.1
--webserver_interface=127.25.107.1
--webserver_port=0
--tserver_master_addrs=127.25.107.62:39107
--builtin_ntp_servers=127.25.107.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:11:44.811416 31282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:44.811740 31282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:44.811832 31282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:44.819581 31282 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:44.819731 31282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:44.819866 31282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.1
I20240501 08:11:44.829303 31282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.107.1
--webserver_port=0
--tserver_master_addrs=127.25.107.62:39107
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.1
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:44.830773 31282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:44.832547 31282 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:44.844538 31290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:44.844600 31287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:44.844573 31288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:44.845289 31282 server_base.cc:1034] running on GCE node
I20240501 08:11:44.846107 31282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:44.847265 31282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:44.848498 31282 hybrid_clock.cc:648] HybridClock initialized: now 1714551104848445 us; error 52 us; skew 500 ppm
I20240501 08:11:44.848824 31282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:44.851220 31282 webserver.cc:460] Webserver started at http://127.25.107.1:38627/ using document root <none> and password file <none>
I20240501 08:11:44.851981 31282 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:44.852082 31282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:44.852367 31282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:44.855105 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "33a32e125b604217b0d23eec5dbe2b73"
format_stamp: "Formatted at 2024-05-01 08:11:44 on dist-test-slave-dv41"
I20240501 08:11:44.855926 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "33a32e125b604217b0d23eec5dbe2b73"
format_stamp: "Formatted at 2024-05-01 08:11:44 on dist-test-slave-dv41"
I20240501 08:11:44.861618 31282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 08:11:44.865479 31296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:44.867038 31282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:44.867214 31282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "33a32e125b604217b0d23eec5dbe2b73"
format_stamp: "Formatted at 2024-05-01 08:11:44 on dist-test-slave-dv41"
I20240501 08:11:44.867419 31282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:44.899551 31282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:44.900478 31282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:44.900765 31282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:44.901798 31282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:44.903460 31282 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:44.903550 31282 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:44.903627 31282 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:44.903705 31282 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:44.933666 31282 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.1:42885
I20240501 08:11:44.933730 31408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.1:42885 every 8 connection(s)
I20240501 08:11:44.935022 31282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:11:44.939365 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 31282
I20240501 08:11:44.939563 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:11:44.942972 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.2:0
--local_ip_for_outbound_sockets=127.25.107.2
--webserver_interface=127.25.107.2
--webserver_port=0
--tserver_master_addrs=127.25.107.62:39107
--builtin_ntp_servers=127.25.107.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:44.948239 31409 heartbeater.cc:347] Connected to a master server at 127.25.107.62:39107
I20240501 08:11:44.948638 31409 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:44.949472 31409 heartbeater.cc:511] Master 127.25.107.62:39107 requested a full tablet report, sending...
I20240501 08:11:44.951910 31223 ts_manager.cc:194] Registered new tserver with Master: 33a32e125b604217b0d23eec5dbe2b73 (127.25.107.1:42885)
I20240501 08:11:44.954052 31223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.1:49153
W20240501 08:11:45.279219 31413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:45.279557 31413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:45.279649 31413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:45.287202 31413 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:45.287338 31413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:45.287465 31413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.2
I20240501 08:11:45.296676 31413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.107.2
--webserver_port=0
--tserver_master_addrs=127.25.107.62:39107
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.2
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:45.298123 31413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:45.300000 31413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:45.312240 31421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.312242 31419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.312283 31418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:45.319967 31413 server_base.cc:1034] running on GCE node
I20240501 08:11:45.320677 31413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:45.321763 31413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:45.322966 31413 hybrid_clock.cc:648] HybridClock initialized: now 1714551105322903 us; error 58 us; skew 500 ppm
I20240501 08:11:45.323302 31413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:45.325655 31413 webserver.cc:460] Webserver started at http://127.25.107.2:42077/ using document root <none> and password file <none>
I20240501 08:11:45.326452 31413 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:45.326575 31413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:45.326901 31413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:45.329849 31413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2fd3e5736eb84190b7cf998c721b8f5b"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-dv41"
I20240501 08:11:45.330687 31413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2fd3e5736eb84190b7cf998c721b8f5b"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-dv41"
I20240501 08:11:45.336129 31413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:45.339711 31427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.341135 31413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:11:45.341301 31413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2fd3e5736eb84190b7cf998c721b8f5b"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-dv41"
I20240501 08:11:45.341499 31413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:45.362684 31413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:45.363605 31413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:45.363865 31413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:45.364872 31413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:45.366437 31413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:45.366530 31413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.366641 31413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:45.366706 31413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.399996 31413 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.2:40611
I20240501 08:11:45.400041 31539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.2:40611 every 8 connection(s)
I20240501 08:11:45.401305 31413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:11:45.403653 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 31413
I20240501 08:11:45.403846 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:11:45.407263 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.3:0
--local_ip_for_outbound_sockets=127.25.107.3
--webserver_interface=127.25.107.3
--webserver_port=0
--tserver_master_addrs=127.25.107.62:39107
--builtin_ntp_servers=127.25.107.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:45.414815 31540 heartbeater.cc:347] Connected to a master server at 127.25.107.62:39107
I20240501 08:11:45.415131 31540 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:45.415931 31540 heartbeater.cc:511] Master 127.25.107.62:39107 requested a full tablet report, sending...
I20240501 08:11:45.417563 31223 ts_manager.cc:194] Registered new tserver with Master: 2fd3e5736eb84190b7cf998c721b8f5b (127.25.107.2:40611)
I20240501 08:11:45.418721 31223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.2:43947
W20240501 08:11:45.742969 31543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:45.743288 31543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:45.743381 31543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:45.751137 31543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:45.751281 31543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:45.751397 31543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.3
I20240501 08:11:45.760661 31543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.107.3
--webserver_port=0
--tserver_master_addrs=127.25.107.62:39107
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.3
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:45.761998 31543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:45.763830 31543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:45.776461 31551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.776563 31549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:45.776412 31548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:45.777371 31543 server_base.cc:1034] running on GCE node
I20240501 08:11:45.778046 31543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:45.779104 31543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:45.780367 31543 hybrid_clock.cc:648] HybridClock initialized: now 1714551105780323 us; error 31 us; skew 500 ppm
I20240501 08:11:45.780758 31543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:45.783257 31543 webserver.cc:460] Webserver started at http://127.25.107.3:40121/ using document root <none> and password file <none>
I20240501 08:11:45.784034 31543 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:45.784157 31543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:45.784497 31543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:45.787343 31543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6a691e8c3e36437da2d2464c9f3be824"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-dv41"
I20240501 08:11:45.788239 31543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6a691e8c3e36437da2d2464c9f3be824"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-dv41"
I20240501 08:11:45.793956 31543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:11:45.800415 31557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.802065 31543 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.006s
I20240501 08:11:45.802297 31543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6a691e8c3e36437da2d2464c9f3be824"
format_stamp: "Formatted at 2024-05-01 08:11:45 on dist-test-slave-dv41"
I20240501 08:11:45.802520 31543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:45.823069 31543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:45.823959 31543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:45.824223 31543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:45.825184 31543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:45.826824 31543 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:45.826916 31543 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.827005 31543 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:45.827060 31543 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:45.856930 31543 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.3:43485
I20240501 08:11:45.856984 31669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.3:43485 every 8 connection(s)
I20240501 08:11:45.858397 31543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:11:45.868598 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 31543
I20240501 08:11:45.868815 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:11:45.872023 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.4:0
--local_ip_for_outbound_sockets=127.25.107.4
--webserver_interface=127.25.107.4
--webserver_port=0
--tserver_master_addrs=127.25.107.62:39107
--builtin_ntp_servers=127.25.107.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:45.876161 31670 heartbeater.cc:347] Connected to a master server at 127.25.107.62:39107
I20240501 08:11:45.876514 31670 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:45.877332 31670 heartbeater.cc:511] Master 127.25.107.62:39107 requested a full tablet report, sending...
I20240501 08:11:45.879029 31223 ts_manager.cc:194] Registered new tserver with Master: 6a691e8c3e36437da2d2464c9f3be824 (127.25.107.3:43485)
I20240501 08:11:45.880117 31223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.3:59133
I20240501 08:11:45.957409 31409 heartbeater.cc:503] Master 127.25.107.62:39107 was elected leader, sending a full tablet report...
W20240501 08:11:46.207867 31673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:46.208185 31673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:46.208317 31673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:46.216025 31673 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:46.216176 31673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:46.216328 31673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.4
I20240501 08:11:46.225587 31673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.107.4
--webserver_port=0
--tserver_master_addrs=127.25.107.62:39107
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.4
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:46.226956 31673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:46.228732 31673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:46.240288 31679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:46.240521 31678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:46.241349 31681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:46.242178 31673 server_base.cc:1034] running on GCE node
I20240501 08:11:46.242846 31673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:46.243809 31673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:46.245020 31673 hybrid_clock.cc:648] HybridClock initialized: now 1714551106244974 us; error 38 us; skew 500 ppm
I20240501 08:11:46.245357 31673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:46.247766 31673 webserver.cc:460] Webserver started at http://127.25.107.4:37431/ using document root <none> and password file <none>
I20240501 08:11:46.248525 31673 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:46.248646 31673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:46.248984 31673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:46.251689 31673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7dd84c0405434d9dbfa3a25eeb4898f7"
format_stamp: "Formatted at 2024-05-01 08:11:46 on dist-test-slave-dv41"
I20240501 08:11:46.252489 31673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7dd84c0405434d9dbfa3a25eeb4898f7"
format_stamp: "Formatted at 2024-05-01 08:11:46 on dist-test-slave-dv41"
I20240501 08:11:46.257952 31673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 08:11:46.261587 31687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:46.263111 31673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 08:11:46.263289 31673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7dd84c0405434d9dbfa3a25eeb4898f7"
format_stamp: "Formatted at 2024-05-01 08:11:46 on dist-test-slave-dv41"
I20240501 08:11:46.263489 31673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:46.290975 31673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:46.291846 31673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:46.292136 31673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:46.293146 31673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:46.294692 31673 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:46.294790 31673 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:46.294904 31673 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:46.294976 31673 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:46.325037 31673 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.4:33259
I20240501 08:11:46.325098 31799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.4:33259 every 8 connection(s)
I20240501 08:11:46.326587 31673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:11:46.333308 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 31673
I20240501 08:11:46.333513 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:11:46.338003 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.5:0
--local_ip_for_outbound_sockets=127.25.107.5
--webserver_interface=127.25.107.5
--webserver_port=0
--tserver_master_addrs=127.25.107.62:39107
--builtin_ntp_servers=127.25.107.20:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:46.350348 31800 heartbeater.cc:347] Connected to a master server at 127.25.107.62:39107
I20240501 08:11:46.350853 31800 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:46.351836 31800 heartbeater.cc:511] Master 127.25.107.62:39107 requested a full tablet report, sending...
I20240501 08:11:46.353351 31223 ts_manager.cc:194] Registered new tserver with Master: 7dd84c0405434d9dbfa3a25eeb4898f7 (127.25.107.4:33259)
I20240501 08:11:46.354303 31223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.4:45677
I20240501 08:11:46.421613 31540 heartbeater.cc:503] Master 127.25.107.62:39107 was elected leader, sending a full tablet report...
W20240501 08:11:46.683961 31803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:46.684280 31803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:46.684397 31803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:46.691948 31803 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:46.692102 31803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:46.692262 31803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.5
I20240501 08:11:46.701516 31803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.107.5
--webserver_port=0
--tserver_master_addrs=127.25.107.62:39107
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.5
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:46.702925 31803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:46.704737 31803 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:46.717089 31808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:46.717147 31809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:46.717128 31811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:46.718119 31803 server_base.cc:1034] running on GCE node
I20240501 08:11:46.718750 31803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:46.719744 31803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:46.720957 31803 hybrid_clock.cc:648] HybridClock initialized: now 1714551106720906 us; error 42 us; skew 500 ppm
I20240501 08:11:46.721302 31803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:46.723958 31803 webserver.cc:460] Webserver started at http://127.25.107.5:43781/ using document root <none> and password file <none>
I20240501 08:11:46.724792 31803 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:46.724910 31803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:46.725267 31803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:46.728137 31803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "850720c8282a46d1b73e995af6cc22a8"
format_stamp: "Formatted at 2024-05-01 08:11:46 on dist-test-slave-dv41"
I20240501 08:11:46.728960 31803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "850720c8282a46d1b73e995af6cc22a8"
format_stamp: "Formatted at 2024-05-01 08:11:46 on dist-test-slave-dv41"
I20240501 08:11:46.734678 31803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 08:11:46.738405 31817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:46.739959 31803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:11:46.740252 31803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "850720c8282a46d1b73e995af6cc22a8"
format_stamp: "Formatted at 2024-05-01 08:11:46 on dist-test-slave-dv41"
I20240501 08:11:46.740489 31803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:46.759562 31803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:46.760469 31803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:46.760768 31803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:46.761809 31803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:46.763424 31803 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:46.763517 31803 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:46.763602 31803 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:46.763655 31803 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:46.793458 31803 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.5:46157
I20240501 08:11:46.793512 31929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.5:46157 every 8 connection(s)
I20240501 08:11:46.794816 31803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:11:46.801455 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 31803
I20240501 08:11:46.801725 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:11:46.810371 31930 heartbeater.cc:347] Connected to a master server at 127.25.107.62:39107
I20240501 08:11:46.810712 31930 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:46.811529 31930 heartbeater.cc:511] Master 127.25.107.62:39107 requested a full tablet report, sending...
I20240501 08:11:46.813011 31223 ts_manager.cc:194] Registered new tserver with Master: 850720c8282a46d1b73e995af6cc22a8 (127.25.107.5:46157)
I20240501 08:11:46.813912 31223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.5:35445
I20240501 08:11:46.818645 26028 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:11:46.835435 31223 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38088:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 08:11:46.883397 31670 heartbeater.cc:503] Master 127.25.107.62:39107 was elected leader, sending a full tablet report...
I20240501 08:11:46.892208 31865 tablet_service.cc:1467] Processing CreateTablet for tablet b95de26024144baabb287d66ad40a79f (DEFAULT_TABLE table=TestTable [id=f7e6f55409b04fcc8acd5aee0d6e9d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.892643 31605 tablet_service.cc:1467] Processing CreateTablet for tablet b95de26024144baabb287d66ad40a79f (DEFAULT_TABLE table=TestTable [id=f7e6f55409b04fcc8acd5aee0d6e9d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.893640 31865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b95de26024144baabb287d66ad40a79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.893973 31605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b95de26024144baabb287d66ad40a79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.898026 31344 tablet_service.cc:1467] Processing CreateTablet for tablet b95de26024144baabb287d66ad40a79f (DEFAULT_TABLE table=TestTable [id=f7e6f55409b04fcc8acd5aee0d6e9d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.899569 31344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b95de26024144baabb287d66ad40a79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.904893 31948 tablet_bootstrap.cc:492] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8: Bootstrap starting.
I20240501 08:11:46.908763 31949 tablet_bootstrap.cc:492] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824: Bootstrap starting.
I20240501 08:11:46.909252 31948 tablet_bootstrap.cc:654] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.910759 31948 log.cc:824] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:46.912284 31949 tablet_bootstrap.cc:654] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.913594 31949 log.cc:824] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:46.914076 31948 tablet_bootstrap.cc:492] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8: No bootstrap required, opened a new log
I20240501 08:11:46.914423 31948 ts_tablet_manager.cc:1367] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240501 08:11:46.916637 31949 tablet_bootstrap.cc:492] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824: No bootstrap required, opened a new log
I20240501 08:11:46.917026 31949 ts_tablet_manager.cc:1367] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.008s
I20240501 08:11:46.919034 31951 tablet_bootstrap.cc:492] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73: Bootstrap starting.
I20240501 08:11:46.921892 31951 tablet_bootstrap.cc:654] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.922958 31951 log.cc:824] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:46.923519 31948 raft_consensus.cc:348] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a691e8c3e36437da2d2464c9f3be824" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43485 } } peers { permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 } } peers { permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 } }
I20240501 08:11:46.923978 31948 raft_consensus.cc:374] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.924089 31948 raft_consensus.cc:729] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850720c8282a46d1b73e995af6cc22a8, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.924918 31948 consensus_queue.cc:260] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a691e8c3e36437da2d2464c9f3be824" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43485 } } peers { permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 } } peers { permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 } }
I20240501 08:11:46.925487 31949 raft_consensus.cc:348] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a691e8c3e36437da2d2464c9f3be824" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43485 } } peers { permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 } } peers { permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 } }
I20240501 08:11:46.925899 31949 raft_consensus.cc:374] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.926002 31949 raft_consensus.cc:729] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a691e8c3e36437da2d2464c9f3be824, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.926182 31930 heartbeater.cc:503] Master 127.25.107.62:39107 was elected leader, sending a full tablet report...
I20240501 08:11:46.926774 31948 ts_tablet_manager.cc:1398] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20240501 08:11:46.926815 31949 consensus_queue.cc:260] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a691e8c3e36437da2d2464c9f3be824" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43485 } } peers { permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 } } peers { permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 } }
I20240501 08:11:46.928617 31949 ts_tablet_manager.cc:1398] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20240501 08:11:46.931376 31951 tablet_bootstrap.cc:492] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73: No bootstrap required, opened a new log
I20240501 08:11:46.931707 31951 ts_tablet_manager.cc:1367] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240501 08:11:46.933081 31955 raft_consensus.cc:482] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:46.933380 31955 raft_consensus.cc:504] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a691e8c3e36437da2d2464c9f3be824" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43485 } } peers { permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 } } peers { permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 } }
I20240501 08:11:46.942032 31951 raft_consensus.cc:348] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a691e8c3e36437da2d2464c9f3be824" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43485 } } peers { permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 } } peers { permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 } }
I20240501 08:11:46.942636 31951 raft_consensus.cc:374] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.942759 31951 raft_consensus.cc:729] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a32e125b604217b0d23eec5dbe2b73, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.943758 31951 consensus_queue.cc:260] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a691e8c3e36437da2d2464c9f3be824" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43485 } } peers { permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 } } peers { permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 } }
I20240501 08:11:46.944981 31885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b95de26024144baabb287d66ad40a79f" candidate_uuid: "6a691e8c3e36437da2d2464c9f3be824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850720c8282a46d1b73e995af6cc22a8" is_pre_election: true
I20240501 08:11:46.945515 31885 raft_consensus.cc:2454] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a691e8c3e36437da2d2464c9f3be824 in term 0.
I20240501 08:11:46.945775 31951 ts_tablet_manager.cc:1398] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20240501 08:11:46.946365 31955 leader_election.cc:288] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 850720c8282a46d1b73e995af6cc22a8 (127.25.107.5:46157), 33a32e125b604217b0d23eec5dbe2b73 (127.25.107.1:42885)
I20240501 08:11:46.946491 31558 leader_election.cc:302] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a691e8c3e36437da2d2464c9f3be824, 850720c8282a46d1b73e995af6cc22a8; no voters:
I20240501 08:11:46.947278 31955 raft_consensus.cc:2781] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:46.947427 31955 raft_consensus.cc:482] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:46.947539 31955 raft_consensus.cc:3037] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.953716 31955 raft_consensus.cc:504] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a691e8c3e36437da2d2464c9f3be824" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43485 } } peers { permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 } } peers { permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 } }
I20240501 08:11:46.954597 31955 leader_election.cc:288] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [CANDIDATE]: Term 1 election: Requested vote from peers 850720c8282a46d1b73e995af6cc22a8 (127.25.107.5:46157), 33a32e125b604217b0d23eec5dbe2b73 (127.25.107.1:42885)
I20240501 08:11:46.955178 31885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b95de26024144baabb287d66ad40a79f" candidate_uuid: "6a691e8c3e36437da2d2464c9f3be824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850720c8282a46d1b73e995af6cc22a8"
I20240501 08:11:46.955411 31885 raft_consensus.cc:3037] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.955647 31363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b95de26024144baabb287d66ad40a79f" candidate_uuid: "6a691e8c3e36437da2d2464c9f3be824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a32e125b604217b0d23eec5dbe2b73"
I20240501 08:11:46.955721 31364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b95de26024144baabb287d66ad40a79f" candidate_uuid: "6a691e8c3e36437da2d2464c9f3be824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a32e125b604217b0d23eec5dbe2b73" is_pre_election: true
I20240501 08:11:46.956105 31363 raft_consensus.cc:3037] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.958168 31885 raft_consensus.cc:2454] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a691e8c3e36437da2d2464c9f3be824 in term 1.
I20240501 08:11:46.958396 31363 raft_consensus.cc:2454] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a691e8c3e36437da2d2464c9f3be824 in term 1.
I20240501 08:11:46.958694 31558 leader_election.cc:302] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a691e8c3e36437da2d2464c9f3be824, 850720c8282a46d1b73e995af6cc22a8; no voters:
I20240501 08:11:46.959017 31955 raft_consensus.cc:2781] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:46.959435 31955 raft_consensus.cc:686] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [term 1 LEADER]: Becoming Leader. State: Replica: 6a691e8c3e36437da2d2464c9f3be824, State: Running, Role: LEADER
I20240501 08:11:46.959900 31955 consensus_queue.cc:237] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a691e8c3e36437da2d2464c9f3be824" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43485 } } peers { permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 } } peers { permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 } }
I20240501 08:11:46.964179 31223 catalog_manager.cc:5484] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a691e8c3e36437da2d2464c9f3be824 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "6a691e8c3e36437da2d2464c9f3be824" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a691e8c3e36437da2d2464c9f3be824" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43485 } } peers { permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 } } peers { permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 } } }
I20240501 08:11:46.993932 26028 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:11:46.996616 26028 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 33a32e125b604217b0d23eec5dbe2b73 to finish bootstrapping
I20240501 08:11:47.005842 26028 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2fd3e5736eb84190b7cf998c721b8f5b to finish bootstrapping
I20240501 08:11:47.014765 26028 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6a691e8c3e36437da2d2464c9f3be824 to finish bootstrapping
I20240501 08:11:47.022598 26028 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7dd84c0405434d9dbfa3a25eeb4898f7 to finish bootstrapping
I20240501 08:11:47.031412 26028 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 850720c8282a46d1b73e995af6cc22a8 to finish bootstrapping
W20240501 08:11:47.046487 31931 tablet.cc:2369] T b95de26024144baabb287d66ad40a79f P 850720c8282a46d1b73e995af6cc22a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:11:47.110697 31671 tablet.cc:2369] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:11:47.188338 31410 tablet.cc:2369] T b95de26024144baabb287d66ad40a79f P 33a32e125b604217b0d23eec5dbe2b73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:47.356823 31800 heartbeater.cc:503] Master 127.25.107.62:39107 was elected leader, sending a full tablet report...
I20240501 08:11:47.432811 31955 consensus_queue.cc:1035] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a32e125b604217b0d23eec5dbe2b73" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:47.440670 31955 consensus_queue.cc:1035] T b95de26024144baabb287d66ad40a79f P 6a691e8c3e36437da2d2464c9f3be824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "850720c8282a46d1b73e995af6cc22a8" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:47.531409 31475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:47.532431 31605 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:47.534994 31344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:47.536866 31735 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:47.541096 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:47.710860 31605 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:47.711272 31475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:47.714365 31344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:47.723775 31735 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:47.726096 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.362329 31605 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:48.378540 31475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.381835 31344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.391566 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.392683 31735 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.554594 31475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.560019 31735 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.561834 31344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.562108 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.562528 31605 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:48.746572 31475 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.753057 31344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.761859 31735 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.763296 31865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:48.789083 31605 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:48.930312 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 31282
I20240501 08:11:48.931778 31404 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:49.027899 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 31282
I20240501 08:11:49.044308 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 31413
I20240501 08:11:49.045547 31535 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:49.139055 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 31413
I20240501 08:11:49.155227 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 31543
I20240501 08:11:49.156328 31664 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:49.254848 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 31543
I20240501 08:11:49.270470 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 31673
I20240501 08:11:49.271541 31794 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:49.362231 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 31673
W20240501 08:11:49.376982 31207 connection.cc:535] server connection from 127.25.107.4:45677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:11:49.377128 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 31803
I20240501 08:11:49.378338 31924 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:49.473670 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 31803
W20240501 08:11:49.488354 31210 connection.cc:535] server connection from 127.25.107.5:35445 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:11:49.488582 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 31192
I20240501 08:11:49.489742 31253 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:49.603346 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 31192
2024-05-01T08:11:49Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (5739 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5740 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240501 08:11:49.645479 26028 test_util.cc:268] Using random seed: 165082254
I20240501 08:11:49.647384 26028 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:11:49.647459 26028 ts_itest-base.cc:113] --------------
I20240501 08:11:49.647511 26028 ts_itest-base.cc:114] 7 tablet servers
I20240501 08:11:49.647549 26028 ts_itest-base.cc:115] 2 replicas per TS
I20240501 08:11:49.647598 26028 ts_itest-base.cc:116] --------------
2024-05-01T08:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:11:49Z Disabled control of system clock
I20240501 08:11:49.673933 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.107.62:43837
--webserver_interface=127.25.107.62
--webserver_port=0
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.107.62:43837
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 08:11:50.013794 32156 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 08:11:50.014189 32156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:50.014312 32156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:50.014417 32156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:50.022075 32156 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:50.022221 32156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:11:50.022336 32156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:50.022397 32156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:11:50.022485 32156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:11:50.031903 32156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.25.107.62:43837
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.107.62:43837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.107.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:50.033378 32156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:50.035269 32156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:50.046329 32162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:50.046550 32161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:50.046370 32164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:50.047308 32156 server_base.cc:1034] running on GCE node
I20240501 08:11:50.048096 32156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:50.049612 32156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:50.050833 32156 hybrid_clock.cc:648] HybridClock initialized: now 1714551110050755 us; error 59 us; skew 500 ppm
I20240501 08:11:50.051193 32156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:50.053665 32156 webserver.cc:460] Webserver started at http://127.25.107.62:41107/ using document root <none> and password file <none>
I20240501 08:11:50.054484 32156 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:50.054627 32156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:50.054934 32156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:50.057794 32156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bce668f82a0c4f169df6570a5b6df5a5"
format_stamp: "Formatted at 2024-05-01 08:11:50 on dist-test-slave-dv41"
I20240501 08:11:50.058641 32156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bce668f82a0c4f169df6570a5b6df5a5"
format_stamp: "Formatted at 2024-05-01 08:11:50 on dist-test-slave-dv41"
I20240501 08:11:50.064270 32156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20240501 08:11:50.068169 32170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:50.069628 32156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 08:11:50.069816 32156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bce668f82a0c4f169df6570a5b6df5a5"
format_stamp: "Formatted at 2024-05-01 08:11:50 on dist-test-slave-dv41"
I20240501 08:11:50.070019 32156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:50.095134 32156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:50.096065 32156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:50.096381 32156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:50.114627 32156 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.62:43837
I20240501 08:11:50.114660 32221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.62:43837 every 8 connection(s)
I20240501 08:11:50.116114 32156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:11:50.120160 32222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:50.123502 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 32156
I20240501 08:11:50.123735 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:11:50.133232 32222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5: Bootstrap starting.
I20240501 08:11:50.136675 32222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:50.138316 32222 log.cc:824] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:50.141162 32222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5: No bootstrap required, opened a new log
I20240501 08:11:50.147935 32222 raft_consensus.cc:348] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce668f82a0c4f169df6570a5b6df5a5" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 43837 } }
I20240501 08:11:50.148329 32222 raft_consensus.cc:374] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:50.148437 32222 raft_consensus.cc:729] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce668f82a0c4f169df6570a5b6df5a5, State: Initialized, Role: FOLLOWER
I20240501 08:11:50.149091 32222 consensus_queue.cc:260] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce668f82a0c4f169df6570a5b6df5a5" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 43837 } }
I20240501 08:11:50.149302 32222 raft_consensus.cc:388] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:50.149431 32222 raft_consensus.cc:482] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:50.149581 32222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:50.151775 32222 raft_consensus.cc:504] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce668f82a0c4f169df6570a5b6df5a5" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 43837 } }
I20240501 08:11:50.152345 32222 leader_election.cc:302] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bce668f82a0c4f169df6570a5b6df5a5; no voters:
I20240501 08:11:50.152983 32222 leader_election.cc:288] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:50.153158 32227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:50.154196 32227 raft_consensus.cc:686] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [term 1 LEADER]: Becoming Leader. State: Replica: bce668f82a0c4f169df6570a5b6df5a5, State: Running, Role: LEADER
I20240501 08:11:50.154961 32227 consensus_queue.cc:237] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce668f82a0c4f169df6570a5b6df5a5" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 43837 } }
I20240501 08:11:50.155277 32222 sys_catalog.cc:564] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:50.157968 32228 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bce668f82a0c4f169df6570a5b6df5a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce668f82a0c4f169df6570a5b6df5a5" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 43837 } } }
I20240501 08:11:50.158378 32229 sys_catalog.cc:455] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bce668f82a0c4f169df6570a5b6df5a5. Latest consensus state: current_term: 1 leader_uuid: "bce668f82a0c4f169df6570a5b6df5a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bce668f82a0c4f169df6570a5b6df5a5" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 43837 } } }
I20240501 08:11:50.158617 32229 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:50.158896 32228 sys_catalog.cc:458] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:50.160164 32236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:50.165016 32236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:50.175498 32236 catalog_manager.cc:1347] Generated new cluster ID: 5c42bbd474fe4dfab7a4621aaa648dae
I20240501 08:11:50.175659 32236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:50.194530 32236 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:50.195845 32236 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:50.207062 32236 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bce668f82a0c4f169df6570a5b6df5a5: Generated new TSK 0
I20240501 08:11:50.208324 32236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:50.219477 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.1:0
--local_ip_for_outbound_sockets=127.25.107.1
--webserver_interface=127.25.107.1
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:11:50.555079 32246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:50.555423 32246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:50.555552 32246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:50.563290 32246 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:50.563429 32246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:50.563583 32246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.1
I20240501 08:11:50.572962 32246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.107.1
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.1
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:50.574352 32246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:50.576283 32246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:50.588392 32252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:50.588424 32254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:50.588392 32251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:50.589370 32246 server_base.cc:1034] running on GCE node
I20240501 08:11:50.590010 32246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:50.591063 32246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:50.592293 32246 hybrid_clock.cc:648] HybridClock initialized: now 1714551110592234 us; error 47 us; skew 500 ppm
I20240501 08:11:50.592664 32246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:50.595093 32246 webserver.cc:460] Webserver started at http://127.25.107.1:43561/ using document root <none> and password file <none>
I20240501 08:11:50.595896 32246 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:50.596011 32246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:50.596328 32246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:50.599511 32246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2219c1853e76474799f93fed12cdc12d"
format_stamp: "Formatted at 2024-05-01 08:11:50 on dist-test-slave-dv41"
I20240501 08:11:50.600390 32246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2219c1853e76474799f93fed12cdc12d"
format_stamp: "Formatted at 2024-05-01 08:11:50 on dist-test-slave-dv41"
I20240501 08:11:50.605748 32246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240501 08:11:50.609438 32260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:50.610947 32246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:50.611137 32246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2219c1853e76474799f93fed12cdc12d"
format_stamp: "Formatted at 2024-05-01 08:11:50 on dist-test-slave-dv41"
I20240501 08:11:50.611347 32246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:50.626230 32246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:50.627224 32246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:50.627506 32246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:50.628486 32246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:50.630110 32246 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:50.630213 32246 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:50.630344 32246 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:50.630406 32246 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:50.660789 32246 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.1:33223
I20240501 08:11:50.660863 32372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.1:33223 every 8 connection(s)
I20240501 08:11:50.662124 32246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:11:50.670408 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 32246
I20240501 08:11:50.670624 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:11:50.673991 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.2:0
--local_ip_for_outbound_sockets=127.25.107.2
--webserver_interface=127.25.107.2
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:50.675635 32373 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:11:50.675953 32373 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:50.676744 32373 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:11:50.679090 32187 ts_manager.cc:194] Registered new tserver with Master: 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:11:50.680615 32187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.1:38949
W20240501 08:11:51.004186 32377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:51.004514 32377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:51.004638 32377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:51.012205 32377 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:51.012364 32377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:51.012521 32377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.2
I20240501 08:11:51.021979 32377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.107.2
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.2
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:51.023373 32377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:51.025163 32377 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:51.037566 32383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:51.037562 32382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:51.037506 32385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:51.038391 32377 server_base.cc:1034] running on GCE node
I20240501 08:11:51.039062 32377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:51.040030 32377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:51.041252 32377 hybrid_clock.cc:648] HybridClock initialized: now 1714551111041183 us; error 49 us; skew 500 ppm
I20240501 08:11:51.041586 32377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:51.048599 32377 webserver.cc:460] Webserver started at http://127.25.107.2:39347/ using document root <none> and password file <none>
I20240501 08:11:51.049400 32377 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:51.049511 32377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:51.049816 32377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:51.052560 32377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
format_stamp: "Formatted at 2024-05-01 08:11:51 on dist-test-slave-dv41"
I20240501 08:11:51.053331 32377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
format_stamp: "Formatted at 2024-05-01 08:11:51 on dist-test-slave-dv41"
I20240501 08:11:51.058926 32377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20240501 08:11:51.062757 32391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:51.064285 32377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:11:51.064442 32377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
format_stamp: "Formatted at 2024-05-01 08:11:51 on dist-test-slave-dv41"
I20240501 08:11:51.064635 32377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:51.084261 32377 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:51.085144 32377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:51.085397 32377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:51.086424 32377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:51.088032 32377 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:51.088121 32377 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:51.088207 32377 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:51.088260 32377 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:51.120004 32377 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.2:46875
I20240501 08:11:51.120035 32503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.2:46875 every 8 connection(s)
I20240501 08:11:51.121420 32377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:11:51.122997 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 32377
I20240501 08:11:51.123170 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:11:51.126706 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.3:0
--local_ip_for_outbound_sockets=127.25.107.3
--webserver_interface=127.25.107.3
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:51.136597 32504 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:11:51.136937 32504 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:51.137764 32504 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:11:51.139322 32187 ts_manager.cc:194] Registered new tserver with Master: b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:51.140398 32187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.2:49371
W20240501 08:11:51.456387 32507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:51.456697 32507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:51.456794 32507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:51.464453 32507 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:51.464596 32507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:51.464727 32507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.3
I20240501 08:11:51.473917 32507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.107.3
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.3
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:51.475332 32507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:51.477145 32507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:51.489115 32512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:51.489078 32515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:51.489063 32513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:51.489908 32507 server_base.cc:1034] running on GCE node
I20240501 08:11:51.490604 32507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:51.491592 32507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:51.492818 32507 hybrid_clock.cc:648] HybridClock initialized: now 1714551111492766 us; error 40 us; skew 500 ppm
I20240501 08:11:51.493151 32507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:51.495523 32507 webserver.cc:460] Webserver started at http://127.25.107.3:38585/ using document root <none> and password file <none>
I20240501 08:11:51.496291 32507 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:51.496412 32507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:51.496742 32507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:51.499429 32507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "12e44d7017614f88b3205111d85de021"
format_stamp: "Formatted at 2024-05-01 08:11:51 on dist-test-slave-dv41"
I20240501 08:11:51.500223 32507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "12e44d7017614f88b3205111d85de021"
format_stamp: "Formatted at 2024-05-01 08:11:51 on dist-test-slave-dv41"
I20240501 08:11:51.505532 32507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:11:51.509058 32521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:51.510480 32507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:51.510651 32507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "12e44d7017614f88b3205111d85de021"
format_stamp: "Formatted at 2024-05-01 08:11:51 on dist-test-slave-dv41"
I20240501 08:11:51.510864 32507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:51.529129 32507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:51.530050 32507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:51.530372 32507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:51.531385 32507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:51.533000 32507 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:51.533106 32507 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:51.533222 32507 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:51.533294 32507 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:51.564131 32507 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.3:38795
I20240501 08:11:51.564181 32633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.3:38795 every 8 connection(s)
I20240501 08:11:51.565559 32507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:11:51.567871 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 32507
I20240501 08:11:51.568068 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:11:51.571636 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.4:0
--local_ip_for_outbound_sockets=127.25.107.4
--webserver_interface=127.25.107.4
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:51.580570 32634 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:11:51.580896 32634 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:51.581679 32634 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:11:51.583281 32187 ts_manager.cc:194] Registered new tserver with Master: 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:11:51.584345 32187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.3:39293
I20240501 08:11:51.683763 32373 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
W20240501 08:11:51.911008 32637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:51.911331 32637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:51.911464 32637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:51.919291 32637 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:51.919446 32637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:51.919607 32637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.4
I20240501 08:11:51.929050 32637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.107.4
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.4
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:51.930486 32637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:51.932272 32637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:51.944021 32643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:51.944203 32642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:51.944169 32645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:51.945135 32637 server_base.cc:1034] running on GCE node
I20240501 08:11:51.945720 32637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:51.946758 32637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:51.947952 32637 hybrid_clock.cc:648] HybridClock initialized: now 1714551111947909 us; error 34 us; skew 500 ppm
I20240501 08:11:51.948266 32637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:51.950513 32637 webserver.cc:460] Webserver started at http://127.25.107.4:33225/ using document root <none> and password file <none>
I20240501 08:11:51.951241 32637 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:51.951339 32637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:51.951651 32637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:51.954198 32637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "489550c4b21146918c1c9618be65b4ff"
format_stamp: "Formatted at 2024-05-01 08:11:51 on dist-test-slave-dv41"
I20240501 08:11:51.955013 32637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "489550c4b21146918c1c9618be65b4ff"
format_stamp: "Formatted at 2024-05-01 08:11:51 on dist-test-slave-dv41"
I20240501 08:11:51.960249 32637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 08:11:51.963748 32651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:51.965130 32637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 08:11:51.965276 32637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "489550c4b21146918c1c9618be65b4ff"
format_stamp: "Formatted at 2024-05-01 08:11:51 on dist-test-slave-dv41"
I20240501 08:11:51.965473 32637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:51.993425 32637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:51.994499 32637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:51.994805 32637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:51.995900 32637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:51.997526 32637 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:51.997640 32637 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:51.997763 32637 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:51.997830 32637 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:52.028168 32637 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.4:39729
I20240501 08:11:52.028234 32763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.4:39729 every 8 connection(s)
I20240501 08:11:52.029422 32637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:11:52.032390 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 32637
I20240501 08:11:52.032545 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:11:52.035769 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.5:0
--local_ip_for_outbound_sockets=127.25.107.5
--webserver_interface=127.25.107.5
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:52.042407 32764 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:11:52.042737 32764 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:52.043509 32764 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:11:52.044935 32187 ts_manager.cc:194] Registered new tserver with Master: 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729)
I20240501 08:11:52.045872 32187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.4:33337
I20240501 08:11:52.143144 32504 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
W20240501 08:11:52.372020 32767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:52.372354 32767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:52.372474 32767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:52.380333 32767 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:52.380499 32767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:52.380667 32767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.5
I20240501 08:11:52.390053 32767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.107.5
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.5
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:52.391461 32767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:52.393289 32767 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:52.405405 307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:52.405452 305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:52.405366 304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:52.406121 32767 server_base.cc:1034] running on GCE node
I20240501 08:11:52.406811 32767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:52.407773 32767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:52.408974 32767 hybrid_clock.cc:648] HybridClock initialized: now 1714551112408923 us; error 43 us; skew 500 ppm
I20240501 08:11:52.409305 32767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:52.411543 32767 webserver.cc:460] Webserver started at http://127.25.107.5:35615/ using document root <none> and password file <none>
I20240501 08:11:52.412312 32767 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:52.412428 32767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:52.412740 32767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:52.415591 32767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e"
format_stamp: "Formatted at 2024-05-01 08:11:52 on dist-test-slave-dv41"
I20240501 08:11:52.416473 32767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e"
format_stamp: "Formatted at 2024-05-01 08:11:52 on dist-test-slave-dv41"
I20240501 08:11:52.421916 32767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:11:52.425652 313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:52.427160 32767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:52.427326 32767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e"
format_stamp: "Formatted at 2024-05-01 08:11:52 on dist-test-slave-dv41"
I20240501 08:11:52.427525 32767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:52.443289 32767 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:52.444172 32767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:52.444450 32767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:52.445472 32767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:52.447072 32767 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:52.447176 32767 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:52.447286 32767 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:52.447355 32767 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:52.476126 32767 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.5:42813
I20240501 08:11:52.476179 425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.5:42813 every 8 connection(s)
I20240501 08:11:52.477452 32767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:11:52.485217 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 32767
I20240501 08:11:52.485417 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:11:52.488986 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.6:0
--local_ip_for_outbound_sockets=127.25.107.6
--webserver_interface=127.25.107.6
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:52.490506 426 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:11:52.490882 426 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:52.491824 426 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:11:52.493487 32187 ts_manager.cc:194] Registered new tserver with Master: 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5:42813)
I20240501 08:11:52.494318 32187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.5:38433
I20240501 08:11:52.587224 32634 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
W20240501 08:11:52.829811 429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:52.830150 429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:52.830328 429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:52.838140 429 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:52.838335 429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:52.838500 429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.6
I20240501 08:11:52.847920 429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.107.6
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.6
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:52.849355 429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:52.851292 429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:52.863919 435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:52.863884 437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:52.864492 434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:52.865623 429 server_base.cc:1034] running on GCE node
I20240501 08:11:52.866300 429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:52.867378 429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:52.868580 429 hybrid_clock.cc:648] HybridClock initialized: now 1714551112868516 us; error 58 us; skew 500 ppm
I20240501 08:11:52.868933 429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:52.871436 429 webserver.cc:460] Webserver started at http://127.25.107.6:36029/ using document root <none> and password file <none>
I20240501 08:11:52.872177 429 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:52.872283 429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:52.872576 429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:52.875392 429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "37965efda2a34916b49a2357eab43bf3"
format_stamp: "Formatted at 2024-05-01 08:11:52 on dist-test-slave-dv41"
I20240501 08:11:52.876194 429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "37965efda2a34916b49a2357eab43bf3"
format_stamp: "Formatted at 2024-05-01 08:11:52 on dist-test-slave-dv41"
I20240501 08:11:52.881695 429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 08:11:52.885422 443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:52.887068 429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:52.887235 429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "37965efda2a34916b49a2357eab43bf3"
format_stamp: "Formatted at 2024-05-01 08:11:52 on dist-test-slave-dv41"
I20240501 08:11:52.887449 429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:52.908752 429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:52.909627 429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:52.909916 429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:52.910964 429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:52.912595 429 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:52.912693 429 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:52.912806 429 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:52.912868 429 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:52.944032 429 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.6:44049
I20240501 08:11:52.944080 555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.6:44049 every 8 connection(s)
I20240501 08:11:52.945364 429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:11:52.948482 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 429
I20240501 08:11:52.948657 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:11:52.952374 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.7:0
--local_ip_for_outbound_sockets=127.25.107.7
--webserver_interface=127.25.107.7
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:52.960304 556 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:11:52.960649 556 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:52.961462 556 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:11:52.963049 32187 ts_manager.cc:194] Registered new tserver with Master: 37965efda2a34916b49a2357eab43bf3 (127.25.107.6:44049)
I20240501 08:11:52.964121 32187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.6:59403
I20240501 08:11:53.049226 32764 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
W20240501 08:11:53.295167 559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:53.295501 559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:53.295601 559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:53.303206 559 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:53.303356 559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:53.303474 559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.7
I20240501 08:11:53.312732 559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.107.7
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.7
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:53.314096 559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:53.316056 559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:53.328943 565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:53.329027 567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:53.329027 564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:53.329922 559 server_base.cc:1034] running on GCE node
I20240501 08:11:53.330581 559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:53.331591 559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:53.332803 559 hybrid_clock.cc:648] HybridClock initialized: now 1714551113332739 us; error 57 us; skew 500 ppm
I20240501 08:11:53.333146 559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:53.335438 559 webserver.cc:460] Webserver started at http://127.25.107.7:44091/ using document root <none> and password file <none>
I20240501 08:11:53.336194 559 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:53.336306 559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:53.336623 559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:53.339414 559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "c5e88f3df73d4d4f996304b6df7e6448"
format_stamp: "Formatted at 2024-05-01 08:11:53 on dist-test-slave-dv41"
I20240501 08:11:53.340227 559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "c5e88f3df73d4d4f996304b6df7e6448"
format_stamp: "Formatted at 2024-05-01 08:11:53 on dist-test-slave-dv41"
I20240501 08:11:53.345695 559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240501 08:11:53.349419 573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:53.350991 559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:53.351155 559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c5e88f3df73d4d4f996304b6df7e6448"
format_stamp: "Formatted at 2024-05-01 08:11:53 on dist-test-slave-dv41"
I20240501 08:11:53.351361 559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:53.370427 559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:53.371366 559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:53.371644 559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:53.372644 559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:53.374315 559 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:53.374405 559 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:53.374521 559 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:53.374583 559 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:53.404718 559 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.7:46501
I20240501 08:11:53.404788 685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.7:46501 every 8 connection(s)
I20240501 08:11:53.406069 559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:11:53.413964 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 559
I20240501 08:11:53.414146 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:11:53.419384 686 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:11:53.419715 686 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:53.420457 686 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:11:53.421824 32187 ts_manager.cc:194] Registered new tserver with Master: c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501)
I20240501 08:11:53.422688 32187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.7:45543
I20240501 08:11:53.431187 26028 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 08:11:53.447913 32187 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36726:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 08:11:53.490089 621 tablet_service.cc:1467] Processing CreateTablet for tablet a50cb92ecb554287977b7a980bd4d2b4 (DEFAULT_TABLE table=TestTable [id=befd81a3b75d41868559acd068af1d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.490079 32439 tablet_service.cc:1467] Processing CreateTablet for tablet a50cb92ecb554287977b7a980bd4d2b4 (DEFAULT_TABLE table=TestTable [id=befd81a3b75d41868559acd068af1d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.491634 621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50cb92ecb554287977b7a980bd4d2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.491640 32439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50cb92ecb554287977b7a980bd4d2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.496941 426 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
I20240501 08:11:53.502293 702 tablet_bootstrap.cc:492] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: Bootstrap starting.
I20240501 08:11:53.502982 703 tablet_bootstrap.cc:492] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap starting.
I20240501 08:11:53.505585 702 tablet_bootstrap.cc:654] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.505735 703 tablet_bootstrap.cc:654] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.506707 702 log.cc:824] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:53.506762 703 log.cc:824] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:53.509416 702 tablet_bootstrap.cc:492] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: No bootstrap required, opened a new log
I20240501 08:11:53.509723 702 ts_tablet_manager.cc:1367] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240501 08:11:53.511291 703 tablet_bootstrap.cc:492] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: No bootstrap required, opened a new log
I20240501 08:11:53.511547 703 ts_tablet_manager.cc:1367] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 08:11:53.516078 702 raft_consensus.cc:348] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:11:53.516506 702 raft_consensus.cc:374] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.516619 702 raft_consensus.cc:729] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.517266 702 consensus_queue.cc:260] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:11:53.518043 703 raft_consensus.cc:348] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:11:53.519047 702 ts_tablet_manager.cc:1398] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240501 08:11:53.519100 703 raft_consensus.cc:374] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.519225 703 raft_consensus.cc:729] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e88f3df73d4d4f996304b6df7e6448, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.519845 703 consensus_queue.cc:260] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:11:53.520972 686 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
I20240501 08:11:53.521360 703 ts_tablet_manager.cc:1398] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240501 08:11:53.570333 707 raft_consensus.cc:482] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:53.570627 707 raft_consensus.cc:504] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:11:53.572182 707 leader_election.cc:288] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:53.575752 706 raft_consensus.cc:482] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:53.576006 706 raft_consensus.cc:504] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:11:53.577682 706 leader_election.cc:288] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501)
I20240501 08:11:53.578296 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50cb92ecb554287977b7a980bd4d2b4" candidate_uuid: "c5e88f3df73d4d4f996304b6df7e6448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" is_pre_election: true
I20240501 08:11:53.578745 32459 raft_consensus.cc:2454] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e88f3df73d4d4f996304b6df7e6448 in term 0.
I20240501 08:11:53.579402 575 leader_election.cc:302] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b3e6c8e1eb974b5c9c8b7e109b214576, c5e88f3df73d4d4f996304b6df7e6448; no voters:
I20240501 08:11:53.579761 707 raft_consensus.cc:2781] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:53.579933 707 raft_consensus.cc:482] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:53.580047 707 raft_consensus.cc:3037] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:53.582829 707 raft_consensus.cc:504] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:11:53.583487 707 leader_election.cc:288] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [CANDIDATE]: Term 1 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:53.584033 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50cb92ecb554287977b7a980bd4d2b4" candidate_uuid: "c5e88f3df73d4d4f996304b6df7e6448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:11:53.584218 32459 raft_consensus.cc:3037] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:53.584861 641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50cb92ecb554287977b7a980bd4d2b4" candidate_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e88f3df73d4d4f996304b6df7e6448" is_pre_election: true
I20240501 08:11:53.585322 641 raft_consensus.cc:2379] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3e6c8e1eb974b5c9c8b7e109b214576 in current term 1: Already voted for candidate c5e88f3df73d4d4f996304b6df7e6448 in this term.
I20240501 08:11:53.586338 32392 leader_election.cc:302] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b3e6c8e1eb974b5c9c8b7e109b214576; no voters: c5e88f3df73d4d4f996304b6df7e6448
I20240501 08:11:53.587137 32459 raft_consensus.cc:2454] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e88f3df73d4d4f996304b6df7e6448 in term 1.
I20240501 08:11:53.587312 706 raft_consensus.cc:2726] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:11:53.587531 575 leader_election.cc:302] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b3e6c8e1eb974b5c9c8b7e109b214576, c5e88f3df73d4d4f996304b6df7e6448; no voters:
I20240501 08:11:53.587774 707 raft_consensus.cc:2781] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:53.588160 707 raft_consensus.cc:686] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 LEADER]: Becoming Leader. State: Replica: c5e88f3df73d4d4f996304b6df7e6448, State: Running, Role: LEADER
I20240501 08:11:53.588565 707 consensus_queue.cc:237] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:11:53.592759 32187 catalog_manager.cc:5484] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7). New cstate: current_term: 1 leader_uuid: "c5e88f3df73d4d4f996304b6df7e6448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
W20240501 08:11:53.626013 32505 tablet.cc:2369] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:53.627662 26028 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 08:11:53.629765 26028 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2219c1853e76474799f93fed12cdc12d to finish bootstrapping
I20240501 08:11:53.639796 26028 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b3e6c8e1eb974b5c9c8b7e109b214576 to finish bootstrapping
I20240501 08:11:53.647923 26028 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 12e44d7017614f88b3205111d85de021 to finish bootstrapping
I20240501 08:11:53.656863 26028 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 489550c4b21146918c1c9618be65b4ff to finish bootstrapping
W20240501 08:11:53.657783 687 tablet.cc:2369] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:53.666304 26028 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 92f76e9a8b5c4bc9b49900f5ca6b4f6e to finish bootstrapping
I20240501 08:11:53.675705 26028 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 37965efda2a34916b49a2357eab43bf3 to finish bootstrapping
I20240501 08:11:53.684780 26028 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c5e88f3df73d4d4f996304b6df7e6448 to finish bootstrapping
I20240501 08:11:53.693507 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 32637
I20240501 08:11:53.694844 32759 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:53.777927 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 32637
I20240501 08:11:53.791173 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 32767
I20240501 08:11:53.792241 421 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:53.872594 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 32767
I20240501 08:11:53.885514 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 429
I20240501 08:11:53.886662 551 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:53.967046 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 429
I20240501 08:11:53.967011 556 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
I20240501 08:11:53.979677 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 559
I20240501 08:11:53.980751 681 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:54.074178 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 559
I20240501 08:11:55.524395 721 raft_consensus.cc:482] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:55.524591 721 raft_consensus.cc:504] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:11:55.525238 721 leader_election.cc:288] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501)
W20240501 08:11:55.526316 32392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:46501: connect: Connection refused (error 111)
W20240501 08:11:55.527773 32392 leader_election.cc:334] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501): Network error: Client connection negotiation failed: client connection to 127.25.107.7:46501: connect: Connection refused (error 111)
I20240501 08:11:55.527935 32392 leader_election.cc:302] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b3e6c8e1eb974b5c9c8b7e109b214576; no voters: c5e88f3df73d4d4f996304b6df7e6448
I20240501 08:11:55.528268 721 raft_consensus.cc:2726] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:11:57.172914 724 raft_consensus.cc:482] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:57.173103 724 raft_consensus.cc:504] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:11:57.173650 724 leader_election.cc:288] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501)
W20240501 08:11:57.176064 32392 leader_election.cc:334] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501): Network error: Client connection negotiation failed: client connection to 127.25.107.7:46501: connect: Connection refused (error 111)
I20240501 08:11:57.176187 32392 leader_election.cc:302] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b3e6c8e1eb974b5c9c8b7e109b214576; no voters: c5e88f3df73d4d4f996304b6df7e6448
I20240501 08:11:57.176458 724 raft_consensus.cc:2726] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:11:57.840023 32187 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36726:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:57.852126 32439 tablet_service.cc:1467] Processing CreateTablet for tablet 789da5b22732463da07c34e8716bbb7e (DEFAULT_TABLE table=rebalance_test_table_0 [id=56218deb6dfb40f5896115db6d37a8ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.852674 32439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789da5b22732463da07c34e8716bbb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.867241 735 tablet_bootstrap.cc:492] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576: Bootstrap starting.
I20240501 08:11:57.869422 735 tablet_bootstrap.cc:654] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.871345 735 tablet_bootstrap.cc:492] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576: No bootstrap required, opened a new log
I20240501 08:11:57.871505 735 ts_tablet_manager.cc:1367] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:57.872449 735 raft_consensus.cc:348] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:57.872675 735 raft_consensus.cc:374] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.872813 735 raft_consensus.cc:729] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.873064 735 consensus_queue.cc:260] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:57.875418 735 ts_tablet_manager.cc:1398] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 08:11:57.879308 32568 tablet_service.cc:1467] Processing CreateTablet for tablet 841db634d09c4ef0bdfda733a489af5d (DEFAULT_TABLE table=rebalance_test_table_0 [id=56218deb6dfb40f5896115db6d37a8ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.880789 32568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 841db634d09c4ef0bdfda733a489af5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.882324 32569 tablet_service.cc:1467] Processing CreateTablet for tablet db3c2a3262974c75bba7743192d66d46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56218deb6dfb40f5896115db6d37a8ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.883116 32569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3c2a3262974c75bba7743192d66d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.884490 32306 tablet_service.cc:1467] Processing CreateTablet for tablet 789da5b22732463da07c34e8716bbb7e (DEFAULT_TABLE table=rebalance_test_table_0 [id=56218deb6dfb40f5896115db6d37a8ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.884516 32307 tablet_service.cc:1467] Processing CreateTablet for tablet 841db634d09c4ef0bdfda733a489af5d (DEFAULT_TABLE table=rebalance_test_table_0 [id=56218deb6dfb40f5896115db6d37a8ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.885839 32306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789da5b22732463da07c34e8716bbb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.886308 32307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 841db634d09c4ef0bdfda733a489af5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.887568 32308 tablet_service.cc:1467] Processing CreateTablet for tablet db3c2a3262974c75bba7743192d66d46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=56218deb6dfb40f5896115db6d37a8ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.888113 32308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3c2a3262974c75bba7743192d66d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.900735 738 tablet_bootstrap.cc:492] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: Bootstrap starting.
I20240501 08:11:57.901368 739 tablet_bootstrap.cc:492] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:11:57.903829 738 tablet_bootstrap.cc:654] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.904071 739 tablet_bootstrap.cc:654] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.904852 738 log.cc:824] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:57.904991 739 log.cc:824] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:57.907305 739 tablet_bootstrap.cc:492] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:11:57.907316 738 tablet_bootstrap.cc:492] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: No bootstrap required, opened a new log
I20240501 08:11:57.907542 739 ts_tablet_manager.cc:1367] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 08:11:57.907545 738 ts_tablet_manager.cc:1367] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:11:57.913726 738 raft_consensus.cc:348] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:57.913723 739 raft_consensus.cc:348] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:57.914094 738 raft_consensus.cc:374] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.914086 739 raft_consensus.cc:374] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.914197 738 raft_consensus.cc:729] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.914196 739 raft_consensus.cc:729] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.914853 738 consensus_queue.cc:260] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:57.914860 739 consensus_queue.cc:260] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:57.916724 739 ts_tablet_manager.cc:1398] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240501 08:11:57.916846 738 ts_tablet_manager.cc:1398] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20240501 08:11:57.917243 739 tablet_bootstrap.cc:492] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:11:57.917325 738 tablet_bootstrap.cc:492] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021: Bootstrap starting.
I20240501 08:11:57.918995 739 tablet_bootstrap.cc:654] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.919019 738 tablet_bootstrap.cc:654] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021: Neither blocks nor log segments found. Creating new log.
W20240501 08:11:57.919090 32374 tablet.cc:2369] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:57.921061 738 tablet_bootstrap.cc:492] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021: No bootstrap required, opened a new log
I20240501 08:11:57.921247 738 ts_tablet_manager.cc:1367] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:57.922291 738 raft_consensus.cc:348] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:57.922560 738 raft_consensus.cc:374] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.922703 738 raft_consensus.cc:729] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.922994 738 consensus_queue.cc:260] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:57.923637 738 ts_tablet_manager.cc:1398] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:57.924674 739 tablet_bootstrap.cc:492] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:11:57.924821 739 ts_tablet_manager.cc:1367] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 08:11:57.925827 739 raft_consensus.cc:348] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:57.926002 739 raft_consensus.cc:374] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.926069 739 raft_consensus.cc:729] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.926311 739 consensus_queue.cc:260] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:57.926733 739 ts_tablet_manager.cc:1398] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:57.927070 739 tablet_bootstrap.cc:492] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:11:57.928692 739 tablet_bootstrap.cc:654] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.930648 739 tablet_bootstrap.cc:492] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:11:57.930774 739 ts_tablet_manager.cc:1367] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 08:11:57.931288 739 raft_consensus.cc:348] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:57.931445 739 raft_consensus.cc:374] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.931499 739 raft_consensus.cc:729] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.931696 739 consensus_queue.cc:260] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:57.932142 739 ts_tablet_manager.cc:1398] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:57.939471 743 raft_consensus.cc:482] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:57.939745 743 raft_consensus.cc:504] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:57.941277 743 leader_election.cc:288] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:11:57.947455 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "841db634d09c4ef0bdfda733a489af5d" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e44d7017614f88b3205111d85de021" is_pre_election: true
I20240501 08:11:57.948086 32589 raft_consensus.cc:2454] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 0.
I20240501 08:11:57.948824 32263 leader_election.cc:302] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:11:57.949245 743 raft_consensus.cc:2781] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:57.949388 743 raft_consensus.cc:482] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:57.949474 743 raft_consensus.cc:3037] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.951612 743 raft_consensus.cc:504] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:57.952173 743 leader_election.cc:288] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Requested vote from peers 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:11:57.952816 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "841db634d09c4ef0bdfda733a489af5d" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e44d7017614f88b3205111d85de021"
I20240501 08:11:57.953024 32589 raft_consensus.cc:3037] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.955343 32589 raft_consensus.cc:2454] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 1.
I20240501 08:11:57.955842 32263 leader_election.cc:302] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:11:57.956162 743 raft_consensus.cc:2781] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:57.956601 743 raft_consensus.cc:686] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Leader. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:11:57.957083 743 consensus_queue.cc:237] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:57.961321 32186 catalog_manager.cc:5484] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2219c1853e76474799f93fed12cdc12d (127.25.107.1). New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } }
W20240501 08:11:58.072023 32635 tablet.cc:2369] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:58.087596 742 raft_consensus.cc:482] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:58.087814 742 raft_consensus.cc:504] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:58.089351 742 leader_election.cc:288] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:11:58.096189 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3c2a3262974c75bba7743192d66d46" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2219c1853e76474799f93fed12cdc12d" is_pre_election: true
I20240501 08:11:58.096599 32328 raft_consensus.cc:2454] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 0.
I20240501 08:11:58.097355 32522 leader_election.cc:302] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:11:58.097859 742 raft_consensus.cc:2781] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:58.098008 742 raft_consensus.cc:482] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:58.098119 742 raft_consensus.cc:3037] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.100205 742 raft_consensus.cc:504] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:58.100786 742 leader_election.cc:288] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Requested vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:11:58.101348 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3c2a3262974c75bba7743192d66d46" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2219c1853e76474799f93fed12cdc12d"
I20240501 08:11:58.101533 32328 raft_consensus.cc:3037] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.103677 32328 raft_consensus.cc:2454] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 1.
I20240501 08:11:58.104151 32522 leader_election.cc:302] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:11:58.104467 742 raft_consensus.cc:2781] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:58.104979 742 raft_consensus.cc:686] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Becoming Leader. State: Replica: 12e44d7017614f88b3205111d85de021, State: Running, Role: LEADER
I20240501 08:11:58.105501 742 consensus_queue.cc:237] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:58.109911 32186 catalog_manager.cc:5484] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e44d7017614f88b3205111d85de021 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } }
I20240501 08:11:58.191797 743 raft_consensus.cc:482] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:58.192025 743 raft_consensus.cc:504] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:58.192590 743 leader_election.cc:288] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:58.198333 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789da5b22732463da07c34e8716bbb7e" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" is_pre_election: true
I20240501 08:11:58.198606 32459 raft_consensus.cc:2454] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 0.
I20240501 08:11:58.199210 32262 leader_election.cc:302] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:11:58.199540 743 raft_consensus.cc:2781] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:58.199666 743 raft_consensus.cc:482] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:58.199790 743 raft_consensus.cc:3037] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.201633 743 raft_consensus.cc:504] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:58.202168 743 leader_election.cc:288] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:58.202675 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789da5b22732463da07c34e8716bbb7e" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:11:58.202872 32459 raft_consensus.cc:3037] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.204701 32459 raft_consensus.cc:2454] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 1.
I20240501 08:11:58.205093 32262 leader_election.cc:302] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:11:58.205372 743 raft_consensus.cc:2781] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:58.205531 743 raft_consensus.cc:686] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Leader. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:11:58.205773 743 consensus_queue.cc:237] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:58.208201 32186 catalog_manager.cc:5484] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2219c1853e76474799f93fed12cdc12d (127.25.107.1). New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } }
I20240501 08:11:58.546838 743 consensus_queue.cc:1035] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:58.561707 755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:58.562049 755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:58.569818 755 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 08:11:58.720932 742 consensus_queue.cc:1035] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:58.733453 32459 raft_consensus.cc:1261] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:58.734448 748 consensus_queue.cc:1035] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:58.811563 796 mvcc.cc:205] Tried to move back new op lower bound from 7022801382319611904 to 7022801380172230656. Current Snapshot: MvccSnapshot[applied={T|T < 7022801382319611904}]
INSERT report
rows total: 2000
time total: 200.884 ms
time per row: 0.100442 ms
I20240501 08:11:58.913480 32185 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36726:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:58.922523 32439 tablet_service.cc:1467] Processing CreateTablet for tablet 8bce7a736006455aa01f06e5542022ec (DEFAULT_TABLE table=rebalance_test_table_1 [id=3910e4f153974479aa62dc08cd2c1ac3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.922776 32568 tablet_service.cc:1467] Processing CreateTablet for tablet 8bce7a736006455aa01f06e5542022ec (DEFAULT_TABLE table=rebalance_test_table_1 [id=3910e4f153974479aa62dc08cd2c1ac3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.923040 32439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bce7a736006455aa01f06e5542022ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.923204 32568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bce7a736006455aa01f06e5542022ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.923442 32307 tablet_service.cc:1467] Processing CreateTablet for tablet 2e794c151e6e40dbb612e64853415d9a (DEFAULT_TABLE table=rebalance_test_table_1 [id=3910e4f153974479aa62dc08cd2c1ac3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.923537 32438 tablet_service.cc:1467] Processing CreateTablet for tablet ad5a9e14873742caa50ec838cca1c715 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3910e4f153974479aa62dc08cd2c1ac3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.923815 32307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e794c151e6e40dbb612e64853415d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.924000 32438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5a9e14873742caa50ec838cca1c715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.926877 32569 tablet_service.cc:1467] Processing CreateTablet for tablet ad5a9e14873742caa50ec838cca1c715 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3910e4f153974479aa62dc08cd2c1ac3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.926859 32437 tablet_service.cc:1467] Processing CreateTablet for tablet 2e794c151e6e40dbb612e64853415d9a (DEFAULT_TABLE table=rebalance_test_table_1 [id=3910e4f153974479aa62dc08cd2c1ac3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.927270 32569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5a9e14873742caa50ec838cca1c715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.928138 32437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e794c151e6e40dbb612e64853415d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.934811 812 tablet_bootstrap.cc:492] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021: Bootstrap starting.
I20240501 08:11:58.936882 812 tablet_bootstrap.cc:654] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.940387 815 tablet_bootstrap.cc:492] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: Bootstrap starting.
I20240501 08:11:58.940774 813 tablet_bootstrap.cc:492] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:11:58.942222 812 tablet_bootstrap.cc:492] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021: No bootstrap required, opened a new log
I20240501 08:11:58.942420 812 ts_tablet_manager.cc:1367] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.007s
I20240501 08:11:58.942461 815 tablet_bootstrap.cc:654] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.942839 813 tablet_bootstrap.cc:654] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.943241 812 raft_consensus.cc:348] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:58.943481 812 raft_consensus.cc:374] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.943586 812 raft_consensus.cc:729] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.943872 812 consensus_queue.cc:260] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:58.944504 812 ts_tablet_manager.cc:1398] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:58.944768 812 tablet_bootstrap.cc:492] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: Bootstrap starting.
I20240501 08:11:58.944804 815 tablet_bootstrap.cc:492] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: No bootstrap required, opened a new log
I20240501 08:11:58.944945 815 ts_tablet_manager.cc:1367] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 08:11:58.945786 813 tablet_bootstrap.cc:492] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:11:58.945734 815 raft_consensus.cc:348] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:58.945931 815 raft_consensus.cc:374] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.945936 813 ts_tablet_manager.cc:1367] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:11:58.946010 815 raft_consensus.cc:729] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.946225 815 consensus_queue.cc:260] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:58.946462 812 tablet_bootstrap.cc:654] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.946756 815 ts_tablet_manager.cc:1398] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:58.946718 813 raft_consensus.cc:348] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:58.946920 813 raft_consensus.cc:374] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.947002 813 raft_consensus.cc:729] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.947039 815 tablet_bootstrap.cc:492] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576: Bootstrap starting.
I20240501 08:11:58.947242 813 consensus_queue.cc:260] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:58.947832 813 ts_tablet_manager.cc:1398] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:58.949224 815 tablet_bootstrap.cc:654] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.955267 812 tablet_bootstrap.cc:492] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: No bootstrap required, opened a new log
I20240501 08:11:58.955461 812 ts_tablet_manager.cc:1367] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20240501 08:11:58.955716 815 tablet_bootstrap.cc:492] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576: No bootstrap required, opened a new log
I20240501 08:11:58.955885 815 ts_tablet_manager.cc:1367] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20240501 08:11:58.956259 812 raft_consensus.cc:348] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:58.956454 812 raft_consensus.cc:374] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.956522 812 raft_consensus.cc:729] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.956478 815 raft_consensus.cc:348] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:58.956684 815 raft_consensus.cc:374] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.956764 815 raft_consensus.cc:729] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.956830 812 consensus_queue.cc:260] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:58.957000 815 consensus_queue.cc:260] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:58.957477 815 ts_tablet_manager.cc:1398] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:58.957834 815 tablet_bootstrap.cc:492] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: Bootstrap starting.
I20240501 08:11:58.957991 812 ts_tablet_manager.cc:1398] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 08:11:58.959442 815 tablet_bootstrap.cc:654] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.961642 815 tablet_bootstrap.cc:492] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: No bootstrap required, opened a new log
I20240501 08:11:58.961794 815 ts_tablet_manager.cc:1367] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:58.962574 815 raft_consensus.cc:348] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:58.962769 815 raft_consensus.cc:374] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.962847 815 raft_consensus.cc:729] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.963042 815 consensus_queue.cc:260] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:58.963454 815 ts_tablet_manager.cc:1398] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:59.029456 742 raft_consensus.cc:482] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.029716 742 raft_consensus.cc:504] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.030450 742 leader_election.cc:288] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:59.037266 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5a9e14873742caa50ec838cca1c715" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" is_pre_election: true
I20240501 08:11:59.037539 32459 raft_consensus.cc:2454] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 0.
I20240501 08:11:59.037972 32523 leader_election.cc:302] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:11:59.038316 742 raft_consensus.cc:2781] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.038457 742 raft_consensus.cc:482] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.038564 742 raft_consensus.cc:3037] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.039352 791 raft_consensus.cc:482] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.039512 791 raft_consensus.cc:504] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.040189 791 leader_election.cc:288] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:59.040692 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bce7a736006455aa01f06e5542022ec" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" is_pre_election: true
I20240501 08:11:59.040942 32459 raft_consensus.cc:2454] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 0.
I20240501 08:11:59.041119 742 raft_consensus.cc:504] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.041342 32523 leader_election.cc:302] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:11:59.041603 791 raft_consensus.cc:2781] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.041610 742 leader_election.cc:288] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:59.041800 791 raft_consensus.cc:482] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.041884 791 raft_consensus.cc:3037] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.043756 791 raft_consensus.cc:504] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.044106 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5a9e14873742caa50ec838cca1c715" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:11:59.044230 791 leader_election.cc:288] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:59.044335 32459 raft_consensus.cc:3037] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.044775 32458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bce7a736006455aa01f06e5542022ec" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:11:59.045015 32458 raft_consensus.cc:3037] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.046552 32459 raft_consensus.cc:2454] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 1.
I20240501 08:11:59.046983 32523 leader_election.cc:302] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:11:59.047277 791 raft_consensus.cc:2781] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.047344 32458 raft_consensus.cc:2454] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 1.
I20240501 08:11:59.047435 791 raft_consensus.cc:686] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Becoming Leader. State: Replica: 12e44d7017614f88b3205111d85de021, State: Running, Role: LEADER
I20240501 08:11:59.047704 791 consensus_queue.cc:237] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.048874 32523 leader_election.cc:302] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:11:59.049153 791 raft_consensus.cc:2781] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.049665 791 raft_consensus.cc:686] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Becoming Leader. State: Replica: 12e44d7017614f88b3205111d85de021, State: Running, Role: LEADER
I20240501 08:11:59.049890 791 consensus_queue.cc:237] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.050537 32182 catalog_manager.cc:5484] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e44d7017614f88b3205111d85de021 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } }
I20240501 08:11:59.055197 32182 catalog_manager.cc:5484] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e44d7017614f88b3205111d85de021 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } }
I20240501 08:11:59.085842 802 raft_consensus.cc:482] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.086063 802 raft_consensus.cc:504] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:59.086771 802 leader_election.cc:288] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:59.087420 32458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e794c151e6e40dbb612e64853415d9a" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" is_pre_election: true
I20240501 08:11:59.087658 32458 raft_consensus.cc:2454] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 0.
I20240501 08:11:59.088085 32262 leader_election.cc:302] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:11:59.088361 802 raft_consensus.cc:2781] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.088466 802 raft_consensus.cc:482] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.088554 802 raft_consensus.cc:3037] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.090781 802 raft_consensus.cc:504] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:59.091264 802 leader_election.cc:288] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:11:59.091820 32458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e794c151e6e40dbb612e64853415d9a" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:11:59.092022 32458 raft_consensus.cc:3037] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.093947 32458 raft_consensus.cc:2454] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 1.
I20240501 08:11:59.094398 32262 leader_election.cc:302] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:11:59.094707 802 raft_consensus.cc:2781] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.094867 802 raft_consensus.cc:686] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Leader. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:11:59.095113 802 consensus_queue.cc:237] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:11:59.097839 32182 catalog_manager.cc:5484] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2219c1853e76474799f93fed12cdc12d (127.25.107.1). New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } }
W20240501 08:11:59.465742 826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:59.466116 826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:59.473917 826 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:59.477671 791 consensus_queue.cc:1035] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:59.479997 791 consensus_queue.cc:1035] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 08:11:59.545706 802 consensus_queue.cc:1035] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 192.379 ms
time per row: 0.0961896 ms
I20240501 08:11:59.813933 32185 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36726:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:59.823087 32568 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbd5afc230c41509774812f082f97f0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=821c266bb4a949cf878e5f7a6df80abf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.823590 32568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbd5afc230c41509774812f082f97f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.823627 32307 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbd5afc230c41509774812f082f97f0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=821c266bb4a949cf878e5f7a6df80abf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.823930 32438 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba5cd7c680b457d99ee9c6934bbcd0d (DEFAULT_TABLE table=rebalance_test_table_2 [id=821c266bb4a949cf878e5f7a6df80abf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.824110 32307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbd5afc230c41509774812f082f97f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.824360 32438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba5cd7c680b457d99ee9c6934bbcd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.824532 32308 tablet_service.cc:1467] Processing CreateTablet for tablet f4ec90b4ddce408a81f588fa7546e0dd (DEFAULT_TABLE table=rebalance_test_table_2 [id=821c266bb4a949cf878e5f7a6df80abf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.824885 32308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ec90b4ddce408a81f588fa7546e0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.826709 32306 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba5cd7c680b457d99ee9c6934bbcd0d (DEFAULT_TABLE table=rebalance_test_table_2 [id=821c266bb4a949cf878e5f7a6df80abf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.827090 32306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba5cd7c680b457d99ee9c6934bbcd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.830734 32569 tablet_service.cc:1467] Processing CreateTablet for tablet f4ec90b4ddce408a81f588fa7546e0dd (DEFAULT_TABLE table=rebalance_test_table_2 [id=821c266bb4a949cf878e5f7a6df80abf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.831211 32569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ec90b4ddce408a81f588fa7546e0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.836270 880 tablet_bootstrap.cc:492] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: Bootstrap starting.
I20240501 08:11:59.836585 882 tablet_bootstrap.cc:492] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576: Bootstrap starting.
I20240501 08:11:59.838210 880 tablet_bootstrap.cc:654] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.838428 882 tablet_bootstrap.cc:654] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.843443 880 tablet_bootstrap.cc:492] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: No bootstrap required, opened a new log
I20240501 08:11:59.843638 880 ts_tablet_manager.cc:1367] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 08:11:59.844501 880 raft_consensus.cc:348] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.844753 880 raft_consensus.cc:374] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.844844 880 raft_consensus.cc:729] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.845431 880 consensus_queue.cc:260] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.846050 880 ts_tablet_manager.cc:1398] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:59.846828 881 tablet_bootstrap.cc:492] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:11:59.849035 881 tablet_bootstrap.cc:654] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.851372 882 tablet_bootstrap.cc:492] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576: No bootstrap required, opened a new log
I20240501 08:11:59.851547 882 ts_tablet_manager.cc:1367] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240501 08:11:59.852175 880 tablet_bootstrap.cc:492] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021: Bootstrap starting.
I20240501 08:11:59.852496 882 raft_consensus.cc:348] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:11:59.852712 882 raft_consensus.cc:374] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.852816 882 raft_consensus.cc:729] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.853085 882 consensus_queue.cc:260] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:11:59.853770 882 ts_tablet_manager.cc:1398] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:59.854866 880 tablet_bootstrap.cc:654] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.855589 881 tablet_bootstrap.cc:492] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:11:59.855764 881 ts_tablet_manager.cc:1367] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 08:11:59.856591 881 raft_consensus.cc:348] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.856814 881 raft_consensus.cc:374] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.856907 881 raft_consensus.cc:729] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.857029 880 tablet_bootstrap.cc:492] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021: No bootstrap required, opened a new log
I20240501 08:11:59.857142 881 consensus_queue.cc:260] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.857203 880 ts_tablet_manager.cc:1367] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:11:59.857585 881 ts_tablet_manager.cc:1398] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:59.857899 881 tablet_bootstrap.cc:492] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:11:59.858021 880 raft_consensus.cc:348] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.858287 880 raft_consensus.cc:374] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.858367 880 raft_consensus.cc:729] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.858594 880 consensus_queue.cc:260] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.859542 881 tablet_bootstrap.cc:654] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.859587 880 ts_tablet_manager.cc:1398] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:59.861577 881 tablet_bootstrap.cc:492] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:11:59.861775 881 ts_tablet_manager.cc:1367] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:59.862633 881 raft_consensus.cc:348] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:11:59.862833 881 raft_consensus.cc:374] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.862910 881 raft_consensus.cc:729] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.863147 881 consensus_queue.cc:260] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:11:59.863649 881 ts_tablet_manager.cc:1398] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:59.864022 881 tablet_bootstrap.cc:492] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:11:59.865434 881 tablet_bootstrap.cc:654] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.867489 881 tablet_bootstrap.cc:492] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:11:59.867666 881 ts_tablet_manager.cc:1367] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:59.868333 881 raft_consensus.cc:348] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.868494 881 raft_consensus.cc:374] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.868582 881 raft_consensus.cc:729] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.868803 881 consensus_queue.cc:260] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.869269 881 ts_tablet_manager.cc:1398] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:59.961438 743 raft_consensus.cc:482] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.961658 743 raft_consensus.cc:504] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.962191 743 leader_election.cc:288] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:11:59.962826 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ec90b4ddce408a81f588fa7546e0dd" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e44d7017614f88b3205111d85de021" is_pre_election: true
I20240501 08:11:59.963073 32589 raft_consensus.cc:2454] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 0.
I20240501 08:11:59.963635 32263 leader_election.cc:302] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:11:59.963961 743 raft_consensus.cc:2781] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.964058 743 raft_consensus.cc:482] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.964161 743 raft_consensus.cc:3037] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.966063 743 raft_consensus.cc:504] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.966668 743 leader_election.cc:288] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Requested vote from peers 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:11:59.967242 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ec90b4ddce408a81f588fa7546e0dd" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e44d7017614f88b3205111d85de021"
I20240501 08:11:59.967449 32589 raft_consensus.cc:3037] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.969487 32589 raft_consensus.cc:2454] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 1.
I20240501 08:11:59.970027 32263 leader_election.cc:302] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:11:59.970391 743 raft_consensus.cc:2781] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.970561 743 raft_consensus.cc:686] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Leader. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:11:59.970817 743 consensus_queue.cc:237] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:11:59.973377 32185 catalog_manager.cc:5484] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2219c1853e76474799f93fed12cdc12d (127.25.107.1). New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } }
I20240501 08:12:00.048619 791 raft_consensus.cc:482] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:00.048820 791 raft_consensus.cc:504] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.049351 791 leader_election.cc:288] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:00.049990 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbd5afc230c41509774812f082f97f0" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2219c1853e76474799f93fed12cdc12d" is_pre_election: true
I20240501 08:12:00.050240 32328 raft_consensus.cc:2454] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 0.
I20240501 08:12:00.050746 32522 leader_election.cc:302] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:12:00.051020 791 raft_consensus.cc:2781] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:00.051141 791 raft_consensus.cc:482] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:00.051213 791 raft_consensus.cc:3037] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.053051 791 raft_consensus.cc:504] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.053525 791 leader_election.cc:288] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Requested vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:00.054123 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbd5afc230c41509774812f082f97f0" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2219c1853e76474799f93fed12cdc12d"
I20240501 08:12:00.054368 32328 raft_consensus.cc:3037] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.056330 32328 raft_consensus.cc:2454] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 1.
I20240501 08:12:00.056803 32522 leader_election.cc:302] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:12:00.057097 791 raft_consensus.cc:2781] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:00.057237 791 raft_consensus.cc:686] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Becoming Leader. State: Replica: 12e44d7017614f88b3205111d85de021, State: Running, Role: LEADER
I20240501 08:12:00.057497 791 consensus_queue.cc:237] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.060048 32185 catalog_manager.cc:5484] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e44d7017614f88b3205111d85de021 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } }
I20240501 08:12:00.106159 743 raft_consensus.cc:482] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:00.106416 743 raft_consensus.cc:504] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:00.107025 743 leader_election.cc:288] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:00.107719 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba5cd7c680b457d99ee9c6934bbcd0d" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" is_pre_election: true
I20240501 08:12:00.107954 32459 raft_consensus.cc:2454] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 0.
I20240501 08:12:00.108428 32262 leader_election.cc:302] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:00.108719 743 raft_consensus.cc:2781] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:00.108831 743 raft_consensus.cc:482] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:00.108909 743 raft_consensus.cc:3037] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.110903 743 raft_consensus.cc:504] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:00.111397 743 leader_election.cc:288] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:00.112042 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba5cd7c680b457d99ee9c6934bbcd0d" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:12:00.112246 32459 raft_consensus.cc:3037] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.114557 32459 raft_consensus.cc:2454] T 5ba5cd7c680b457d99ee9c6934bbcd0d P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 1.
I20240501 08:12:00.115079 32262 leader_election.cc:302] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:00.115478 743 raft_consensus.cc:2781] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:00.115656 743 raft_consensus.cc:686] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Leader. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:12:00.116021 743 consensus_queue.cc:237] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:00.119124 32185 catalog_manager.cc:5484] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2219c1853e76474799f93fed12cdc12d (127.25.107.1). New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } }
W20240501 08:12:00.472110 892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:00.472473 892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:00.480170 892 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:00.493986 743 consensus_queue.cc:1035] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 08:12:00.588904 802 consensus_queue.cc:1035] T 5ba5cd7c680b457d99ee9c6934bbcd0d P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:00.618618 791 consensus_queue.cc:1035] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 191.052 ms
time per row: 0.0955262 ms
I20240501 08:12:00.816107 32184 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36726:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:00.826879 32438 tablet_service.cc:1467] Processing CreateTablet for tablet 95a25d38a68a424ebf78f02d2e0750b1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc8097a76e54347ad4bc67576658bc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.827032 32569 tablet_service.cc:1467] Processing CreateTablet for tablet 95a25d38a68a424ebf78f02d2e0750b1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc8097a76e54347ad4bc67576658bc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.827417 32438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a25d38a68a424ebf78f02d2e0750b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.827487 32569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a25d38a68a424ebf78f02d2e0750b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.827759 32307 tablet_service.cc:1467] Processing CreateTablet for tablet ddaae299e96b412282bec452f9b06fa2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc8097a76e54347ad4bc67576658bc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.827895 32439 tablet_service.cc:1467] Processing CreateTablet for tablet 50c58a563cf94527895a675a1e729fe8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc8097a76e54347ad4bc67576658bc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.828147 32307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddaae299e96b412282bec452f9b06fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.828246 32439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c58a563cf94527895a675a1e729fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.835237 32568 tablet_service.cc:1467] Processing CreateTablet for tablet 50c58a563cf94527895a675a1e729fe8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc8097a76e54347ad4bc67576658bc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.835820 32568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c58a563cf94527895a675a1e729fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.836724 32437 tablet_service.cc:1467] Processing CreateTablet for tablet ddaae299e96b412282bec452f9b06fa2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2cc8097a76e54347ad4bc67576658bc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.837163 32437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddaae299e96b412282bec452f9b06fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.839648 947 tablet_bootstrap.cc:492] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: Bootstrap starting.
I20240501 08:12:00.841692 947 tablet_bootstrap.cc:654] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.845713 948 tablet_bootstrap.cc:492] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: Bootstrap starting.
I20240501 08:12:00.847285 949 tablet_bootstrap.cc:492] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:12:00.848306 948 tablet_bootstrap.cc:654] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.849622 947 tablet_bootstrap.cc:492] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: No bootstrap required, opened a new log
I20240501 08:12:00.849992 947 ts_tablet_manager.cc:1367] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240501 08:12:00.850447 949 tablet_bootstrap.cc:654] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.851123 947 raft_consensus.cc:348] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.851411 947 raft_consensus.cc:374] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.851526 947 raft_consensus.cc:729] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.851778 947 consensus_queue.cc:260] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.852090 948 tablet_bootstrap.cc:492] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: No bootstrap required, opened a new log
I20240501 08:12:00.852263 948 ts_tablet_manager.cc:1367] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:12:00.853036 947 ts_tablet_manager.cc:1398] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:00.853057 948 raft_consensus.cc:348] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.853284 948 raft_consensus.cc:374] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.853329 947 tablet_bootstrap.cc:492] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021: Bootstrap starting.
I20240501 08:12:00.853384 948 raft_consensus.cc:729] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.853600 948 consensus_queue.cc:260] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.854138 948 ts_tablet_manager.cc:1398] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:00.854456 948 tablet_bootstrap.cc:492] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576: Bootstrap starting.
I20240501 08:12:00.855618 947 tablet_bootstrap.cc:654] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.856045 948 tablet_bootstrap.cc:654] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.859177 949 tablet_bootstrap.cc:492] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:12:00.859373 949 ts_tablet_manager.cc:1367] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240501 08:12:00.859724 947 tablet_bootstrap.cc:492] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021: No bootstrap required, opened a new log
I20240501 08:12:00.859882 948 tablet_bootstrap.cc:492] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576: No bootstrap required, opened a new log
I20240501 08:12:00.859887 947 ts_tablet_manager.cc:1367] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:12:00.860095 948 ts_tablet_manager.cc:1367] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240501 08:12:00.860248 949 raft_consensus.cc:348] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:00.860466 949 raft_consensus.cc:374] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.860563 949 raft_consensus.cc:729] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.860752 947 raft_consensus.cc:348] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.860795 949 consensus_queue.cc:260] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:00.860954 947 raft_consensus.cc:374] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.860852 948 raft_consensus.cc:348] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.861028 947 raft_consensus.cc:729] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.861251 947 consensus_queue.cc:260] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.861354 949 ts_tablet_manager.cc:1398] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:00.861058 948 raft_consensus.cc:374] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.861645 948 raft_consensus.cc:729] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.861758 947 ts_tablet_manager.cc:1398] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:00.861905 948 consensus_queue.cc:260] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.862591 948 ts_tablet_manager.cc:1398] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:00.863190 948 tablet_bootstrap.cc:492] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576: Bootstrap starting.
I20240501 08:12:00.866881 948 tablet_bootstrap.cc:654] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.868935 948 tablet_bootstrap.cc:492] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576: No bootstrap required, opened a new log
I20240501 08:12:00.869082 948 ts_tablet_manager.cc:1367] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 08:12:00.869717 948 raft_consensus.cc:348] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:00.869884 948 raft_consensus.cc:374] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.869974 948 raft_consensus.cc:729] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.870194 948 consensus_queue.cc:260] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:00.870738 948 ts_tablet_manager.cc:1398] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:00.955344 931 raft_consensus.cc:482] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:00.955560 931 raft_consensus.cc:504] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.956120 931 leader_election.cc:288] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:00.956830 32458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a25d38a68a424ebf78f02d2e0750b1" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" is_pre_election: true
I20240501 08:12:00.957037 32458 raft_consensus.cc:2454] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 0.
I20240501 08:12:00.957473 32523 leader_election.cc:302] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:00.957763 931 raft_consensus.cc:2781] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:00.957872 931 raft_consensus.cc:482] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:00.957942 931 raft_consensus.cc:3037] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.959906 931 raft_consensus.cc:504] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.960397 931 leader_election.cc:288] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:00.960973 32458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a25d38a68a424ebf78f02d2e0750b1" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:12:00.961156 32458 raft_consensus.cc:3037] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.963119 32458 raft_consensus.cc:2454] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 1.
I20240501 08:12:00.963572 32523 leader_election.cc:302] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:00.963891 931 raft_consensus.cc:2781] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:00.964015 931 raft_consensus.cc:686] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Becoming Leader. State: Replica: 12e44d7017614f88b3205111d85de021, State: Running, Role: LEADER
I20240501 08:12:00.964272 931 consensus_queue.cc:237] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:00.967041 32186 catalog_manager.cc:5484] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e44d7017614f88b3205111d85de021 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } }
I20240501 08:12:01.006098 931 raft_consensus.cc:482] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:01.006371 931 raft_consensus.cc:504] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:01.007004 931 leader_election.cc:288] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:01.007613 32458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c58a563cf94527895a675a1e729fe8" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" is_pre_election: true
I20240501 08:12:01.007843 32458 raft_consensus.cc:2454] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 0.
I20240501 08:12:01.008280 32523 leader_election.cc:302] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:01.008581 931 raft_consensus.cc:2781] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:01.008700 931 raft_consensus.cc:482] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:01.008803 931 raft_consensus.cc:3037] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.010757 931 raft_consensus.cc:504] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:01.011273 931 leader_election.cc:288] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:01.011833 32458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c58a563cf94527895a675a1e729fe8" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:12:01.012081 32458 raft_consensus.cc:3037] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.014142 32458 raft_consensus.cc:2454] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 1.
I20240501 08:12:01.014681 32523 leader_election.cc:302] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:01.015056 931 raft_consensus.cc:2781] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:01.015244 931 raft_consensus.cc:686] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Becoming Leader. State: Replica: 12e44d7017614f88b3205111d85de021, State: Running, Role: LEADER
I20240501 08:12:01.015517 931 consensus_queue.cc:237] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:01.018291 32186 catalog_manager.cc:5484] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e44d7017614f88b3205111d85de021 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } }
I20240501 08:12:01.131157 921 raft_consensus.cc:482] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:01.131354 921 raft_consensus.cc:504] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:01.132143 921 leader_election.cc:288] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:01.139510 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddaae299e96b412282bec452f9b06fa2" candidate_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2219c1853e76474799f93fed12cdc12d" is_pre_election: true
I20240501 08:12:01.139818 32328 raft_consensus.cc:2454] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e6c8e1eb974b5c9c8b7e109b214576 in term 0.
I20240501 08:12:01.140259 32392 leader_election.cc:302] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:01.140540 921 raft_consensus.cc:2781] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:01.140684 921 raft_consensus.cc:482] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:01.140761 921 raft_consensus.cc:3037] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.142915 921 raft_consensus.cc:504] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:01.143390 921 leader_election.cc:288] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 1 election: Requested vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:01.143975 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddaae299e96b412282bec452f9b06fa2" candidate_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2219c1853e76474799f93fed12cdc12d"
I20240501 08:12:01.144222 32328 raft_consensus.cc:3037] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.146428 32328 raft_consensus.cc:2454] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e6c8e1eb974b5c9c8b7e109b214576 in term 1.
I20240501 08:12:01.146806 32392 leader_election.cc:302] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:01.147082 921 raft_consensus.cc:2781] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:01.147467 921 raft_consensus.cc:686] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 LEADER]: Becoming Leader. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Running, Role: LEADER
I20240501 08:12:01.147882 921 consensus_queue.cc:237] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:01.151082 32186 catalog_manager.cc:5484] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2). New cstate: current_term: 1 leader_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } }
I20240501 08:12:01.413573 931 consensus_queue.cc:1035] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:01.480405 791 consensus_queue.cc:1035] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:01.552177 962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:01.552531 962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:01.560354 962 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 08:12:01.707998 32328 raft_consensus.cc:1261] T ddaae299e96b412282bec452f9b06fa2 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:01.715600 921 consensus_queue.cc:1035] T ddaae299e96b412282bec452f9b06fa2 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 175.983 ms
time per row: 0.0879917 ms
I20240501 08:12:01.883924 32186 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36726:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:01.892542 32568 tablet_service.cc:1467] Processing CreateTablet for tablet adbf537bdcdf4443a94799340aab4205 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e2d2d2401bec456884eaceee2e5edf67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.892545 32307 tablet_service.cc:1467] Processing CreateTablet for tablet adbf537bdcdf4443a94799340aab4205 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e2d2d2401bec456884eaceee2e5edf67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.893057 32307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbf537bdcdf4443a94799340aab4205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.893065 32568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbf537bdcdf4443a94799340aab4205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.893349 32439 tablet_service.cc:1467] Processing CreateTablet for tablet a9d4354546c040d6b971ed6fc3b02c76 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e2d2d2401bec456884eaceee2e5edf67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.893724 32439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d4354546c040d6b971ed6fc3b02c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.893638 32569 tablet_service.cc:1467] Processing CreateTablet for tablet b4c8611ac7394bbd96cabee647f5c724 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e2d2d2401bec456884eaceee2e5edf67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.894018 32569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c8611ac7394bbd96cabee647f5c724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.900041 32308 tablet_service.cc:1467] Processing CreateTablet for tablet b4c8611ac7394bbd96cabee647f5c724 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e2d2d2401bec456884eaceee2e5edf67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.900522 32308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c8611ac7394bbd96cabee647f5c724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.906728 32306 tablet_service.cc:1467] Processing CreateTablet for tablet a9d4354546c040d6b971ed6fc3b02c76 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e2d2d2401bec456884eaceee2e5edf67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.907167 32306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d4354546c040d6b971ed6fc3b02c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.909601 1020 tablet_bootstrap.cc:492] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:12:01.910933 1022 tablet_bootstrap.cc:492] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576: Bootstrap starting.
I20240501 08:12:01.913467 1022 tablet_bootstrap.cc:654] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.913586 1021 tablet_bootstrap.cc:492] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021: Bootstrap starting.
I20240501 08:12:01.913901 1020 tablet_bootstrap.cc:654] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.915377 1021 tablet_bootstrap.cc:654] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.915702 1022 tablet_bootstrap.cc:492] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576: No bootstrap required, opened a new log
I20240501 08:12:01.915845 1022 ts_tablet_manager.cc:1367] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 08:12:01.916649 1022 raft_consensus.cc:348] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:01.916874 1022 raft_consensus.cc:374] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.916963 1022 raft_consensus.cc:729] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.917197 1022 consensus_queue.cc:260] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:01.917344 1020 tablet_bootstrap.cc:492] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:12:01.917484 1020 ts_tablet_manager.cc:1367] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240501 08:12:01.917819 1022 ts_tablet_manager.cc:1398] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:01.918135 1020 raft_consensus.cc:348] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:12:01.917408 1021 tablet_bootstrap.cc:492] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021: No bootstrap required, opened a new log
I20240501 08:12:01.918319 1020 raft_consensus.cc:374] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.918403 1020 raft_consensus.cc:729] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.918460 1021 ts_tablet_manager.cc:1367] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:12:01.918610 1020 consensus_queue.cc:260] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:12:01.919137 1020 ts_tablet_manager.cc:1398] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:01.919186 1021 raft_consensus.cc:348] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:01.919377 1021 raft_consensus.cc:374] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.919457 1021 raft_consensus.cc:729] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.919477 1020 tablet_bootstrap.cc:492] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:12:01.919658 1021 consensus_queue.cc:260] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:01.920212 1021 ts_tablet_manager.cc:1398] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:01.920454 1021 tablet_bootstrap.cc:492] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021: Bootstrap starting.
I20240501 08:12:01.922015 1021 tablet_bootstrap.cc:654] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.922427 1020 tablet_bootstrap.cc:654] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.927320 1021 tablet_bootstrap.cc:492] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021: No bootstrap required, opened a new log
I20240501 08:12:01.927354 1020 tablet_bootstrap.cc:492] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:12:01.927510 1020 ts_tablet_manager.cc:1367] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 08:12:01.927512 1021 ts_tablet_manager.cc:1367] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20240501 08:12:01.928264 1020 raft_consensus.cc:348] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:01.928267 1021 raft_consensus.cc:348] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:12:01.928488 1021 raft_consensus.cc:374] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.928498 1020 raft_consensus.cc:374] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.928575 1021 raft_consensus.cc:729] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.928582 1020 raft_consensus.cc:729] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.928813 1021 consensus_queue.cc:260] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:12:01.928823 1020 consensus_queue.cc:260] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:01.929347 1020 ts_tablet_manager.cc:1398] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:01.929416 1021 ts_tablet_manager.cc:1398] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 08:12:01.929656 1020 tablet_bootstrap.cc:492] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: Bootstrap starting.
I20240501 08:12:01.931315 1020 tablet_bootstrap.cc:654] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.933318 1020 tablet_bootstrap.cc:492] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: No bootstrap required, opened a new log
I20240501 08:12:01.933463 1020 ts_tablet_manager.cc:1367] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:01.934155 1020 raft_consensus.cc:348] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:01.934374 1020 raft_consensus.cc:374] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.934468 1020 raft_consensus.cc:729] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.934725 1020 consensus_queue.cc:260] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:01.935181 1020 ts_tablet_manager.cc:1398] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:01.962158 999 raft_consensus.cc:482] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:01.962397 999 raft_consensus.cc:504] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:01.962980 999 leader_election.cc:288] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:01.963626 32458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4354546c040d6b971ed6fc3b02c76" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" is_pre_election: true
I20240501 08:12:01.963842 32458 raft_consensus.cc:2454] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 0.
I20240501 08:12:01.964289 32262 leader_election.cc:302] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:01.964579 999 raft_consensus.cc:2781] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:01.964680 999 raft_consensus.cc:482] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:01.964769 999 raft_consensus.cc:3037] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.967062 999 raft_consensus.cc:504] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:01.967566 999 leader_election.cc:288] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:01.968087 32458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4354546c040d6b971ed6fc3b02c76" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:12:01.968288 32458 raft_consensus.cc:3037] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.970197 32458 raft_consensus.cc:2454] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 1.
I20240501 08:12:01.970656 32262 leader_election.cc:302] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:01.970963 999 raft_consensus.cc:2781] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:01.971088 999 raft_consensus.cc:686] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Leader. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:12:01.971345 999 consensus_queue.cc:237] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } }
I20240501 08:12:01.973862 32184 catalog_manager.cc:5484] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2219c1853e76474799f93fed12cdc12d (127.25.107.1). New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } }
I20240501 08:12:02.060925 1008 raft_consensus.cc:482] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:02.061108 1008 raft_consensus.cc:504] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:12:02.061693 1008 leader_election.cc:288] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:02.062274 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbf537bdcdf4443a94799340aab4205" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2219c1853e76474799f93fed12cdc12d" is_pre_election: true
I20240501 08:12:02.062465 32328 raft_consensus.cc:2454] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 0.
I20240501 08:12:02.062839 32522 leader_election.cc:302] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:12:02.063141 1008 raft_consensus.cc:2781] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:02.063278 1008 raft_consensus.cc:482] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:02.063352 1008 raft_consensus.cc:3037] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.065232 1008 raft_consensus.cc:504] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:12:02.065660 1008 leader_election.cc:288] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Requested vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:02.066198 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbf537bdcdf4443a94799340aab4205" candidate_uuid: "12e44d7017614f88b3205111d85de021" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2219c1853e76474799f93fed12cdc12d"
I20240501 08:12:02.066437 32328 raft_consensus.cc:3037] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.068455 32328 raft_consensus.cc:2454] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e44d7017614f88b3205111d85de021 in term 1.
I20240501 08:12:02.068866 32522 leader_election.cc:302] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:12:02.069159 1008 raft_consensus.cc:2781] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:02.069309 1008 raft_consensus.cc:686] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Becoming Leader. State: Replica: 12e44d7017614f88b3205111d85de021, State: Running, Role: LEADER
I20240501 08:12:02.069582 1008 consensus_queue.cc:237] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } }
I20240501 08:12:02.072314 32184 catalog_manager.cc:5484] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e44d7017614f88b3205111d85de021 (127.25.107.3). New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } }
I20240501 08:12:02.105705 999 raft_consensus.cc:482] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:02.105962 999 raft_consensus.cc:504] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:02.106564 999 leader_election.cc:288] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:12:02.107348 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c8611ac7394bbd96cabee647f5c724" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e44d7017614f88b3205111d85de021" is_pre_election: true
I20240501 08:12:02.107595 32589 raft_consensus.cc:2454] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 0.
I20240501 08:12:02.108120 32263 leader_election.cc:302] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:12:02.108451 999 raft_consensus.cc:2781] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:02.108567 999 raft_consensus.cc:482] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:02.108701 999 raft_consensus.cc:3037] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.110852 999 raft_consensus.cc:504] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:02.111413 999 leader_election.cc:288] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Requested vote from peers 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:12:02.112030 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c8611ac7394bbd96cabee647f5c724" candidate_uuid: "2219c1853e76474799f93fed12cdc12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e44d7017614f88b3205111d85de021"
I20240501 08:12:02.112254 32589 raft_consensus.cc:3037] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.114523 32589 raft_consensus.cc:2454] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2219c1853e76474799f93fed12cdc12d in term 1.
I20240501 08:12:02.114959 32263 leader_election.cc:302] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 2219c1853e76474799f93fed12cdc12d; no voters:
I20240501 08:12:02.115239 999 raft_consensus.cc:2781] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:02.115466 999 raft_consensus.cc:686] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Leader. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:12:02.115718 999 consensus_queue.cc:237] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } }
I20240501 08:12:02.118695 32184 catalog_manager.cc:5484] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2219c1853e76474799f93fed12cdc12d (127.25.107.1). New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } }
I20240501 08:12:02.212586 1013 raft_consensus.cc:482] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:02.212790 1013 raft_consensus.cc:504] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:02.213384 1013 leader_election.cc:288] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501)
W20240501 08:12:02.214207 32392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:46501: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:12:02.215612 32392 leader_election.cc:334] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501): Network error: Client connection negotiation failed: client connection to 127.25.107.7:46501: connect: Connection refused (error 111)
I20240501 08:12:02.215754 32392 leader_election.cc:302] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b3e6c8e1eb974b5c9c8b7e109b214576; no voters: c5e88f3df73d4d4f996304b6df7e6448
I20240501 08:12:02.216014 1013 raft_consensus.cc:2726] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:12:02.432958 999 consensus_queue.cc:1035] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:02.481372 1032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:02.481719 1032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:02.489249 1032 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 08:12:02.555780 1008 consensus_queue.cc:1035] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:02.597393 802 consensus_queue.cc:1035] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 199.908 ms
time per row: 0.0999541 ms
I20240501 08:12:02.829192 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.4:39729
--local_ip_for_outbound_sockets=127.25.107.4
--tserver_master_addrs=127.25.107.62:43837
--webserver_port=33225
--webserver_interface=127.25.107.4
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:12:03.182319 1080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:03.182646 1080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:03.182775 1080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:03.191012 1080 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:12:03.191237 1080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:03.191416 1080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.4
I20240501 08:12:03.203807 1080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.4:39729
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.107.4
--webserver_port=33225
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.4
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:03.205212 1080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:03.207116 1080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:03.221177 1086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:03.221249 1085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:03.221488 1088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:03.222362 1080 server_base.cc:1034] running on GCE node
I20240501 08:12:03.223016 1080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:03.224148 1080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:03.225363 1080 hybrid_clock.cc:648] HybridClock initialized: now 1714551123225290 us; error 72 us; skew 500 ppm
I20240501 08:12:03.225709 1080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:03.228168 1080 webserver.cc:460] Webserver started at http://127.25.107.4:33225/ using document root <none> and password file <none>
I20240501 08:12:03.229035 1080 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:03.229182 1080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:03.235073 1080 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 08:12:03.238147 1094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.239708 1080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:03.239888 1080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "489550c4b21146918c1c9618be65b4ff"
format_stamp: "Formatted at 2024-05-01 08:11:51 on dist-test-slave-dv41"
I20240501 08:12:03.240783 1080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:03.262079 1080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:03.262984 1080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:03.263304 1080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:03.264277 1080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:03.265863 1080 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:03.265946 1080 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.266074 1080 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:03.266127 1080 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.297063 1080 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.4:39729
I20240501 08:12:03.297102 1206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.4:39729 every 8 connection(s)
I20240501 08:12:03.298601 1080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:12:03.300173 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 1080
I20240501 08:12:03.300912 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.5:42813
--local_ip_for_outbound_sockets=127.25.107.5
--tserver_master_addrs=127.25.107.62:43837
--webserver_port=35615
--webserver_interface=127.25.107.5
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:12:03.313644 1207 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:12:03.313966 1207 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:03.314788 1207 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:12:03.316426 32182 ts_manager.cc:194] Re-registered known tserver with Master: 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729)
I20240501 08:12:03.317315 32182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.4:52235
W20240501 08:12:03.645695 1211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:03.646068 1211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:03.646215 1211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:03.654608 1211 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:12:03.654762 1211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:03.654918 1211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.5
I20240501 08:12:03.664407 1211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.5:42813
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.107.5
--webserver_port=35615
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.5
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:03.665820 1211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:03.667738 1211 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:03.681649 1216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:03.681915 1219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:03.681788 1217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:03.682840 1211 server_base.cc:1034] running on GCE node
I20240501 08:12:03.683490 1211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:03.684494 1211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:03.685698 1211 hybrid_clock.cc:648] HybridClock initialized: now 1714551123685656 us; error 40 us; skew 500 ppm
I20240501 08:12:03.686031 1211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:03.688563 1211 webserver.cc:460] Webserver started at http://127.25.107.5:35615/ using document root <none> and password file <none>
I20240501 08:12:03.689498 1211 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:03.689620 1211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:03.695044 1211 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:03.697888 1225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.699504 1211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 08:12:03.699678 1211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e"
format_stamp: "Formatted at 2024-05-01 08:11:52 on dist-test-slave-dv41"
I20240501 08:12:03.700580 1211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:03.742787 1211 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:03.743702 1211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:03.743999 1211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:03.745180 1211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:03.747081 1211 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:03.747159 1211 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.747277 1211 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:03.747324 1211 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.778445 1211 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.5:42813
I20240501 08:12:03.778503 1337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.5:42813 every 8 connection(s)
I20240501 08:12:03.780004 1211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:12:03.784057 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 1211
I20240501 08:12:03.784788 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.6:44049
--local_ip_for_outbound_sockets=127.25.107.6
--tserver_master_addrs=127.25.107.62:43837
--webserver_port=36029
--webserver_interface=127.25.107.6
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:12:03.793983 1338 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:12:03.794410 1338 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:03.795352 1338 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:12:03.797037 32182 ts_manager.cc:194] Re-registered known tserver with Master: 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5:42813)
I20240501 08:12:03.798202 32182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.5:37321
W20240501 08:12:04.133255 1341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:04.133695 1341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:04.133859 1341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:04.142158 1341 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:12:04.142357 1341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:04.142529 1341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.6
I20240501 08:12:04.152412 1341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.6:44049
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.107.6
--webserver_port=36029
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.6
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:04.153774 1341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:04.155705 1341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:04.168174 1346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:04.168202 1347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:04.168256 1349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:04.169268 1341 server_base.cc:1034] running on GCE node
I20240501 08:12:04.169893 1341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:04.170926 1341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:04.172147 1341 hybrid_clock.cc:648] HybridClock initialized: now 1714551124172086 us; error 52 us; skew 500 ppm
I20240501 08:12:04.172479 1341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:04.174909 1341 webserver.cc:460] Webserver started at http://127.25.107.6:36029/ using document root <none> and password file <none>
I20240501 08:12:04.175746 1341 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:04.175880 1341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:04.181756 1341 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:12:04.184844 1355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:04.186448 1341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:04.186620 1341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "37965efda2a34916b49a2357eab43bf3"
format_stamp: "Formatted at 2024-05-01 08:11:52 on dist-test-slave-dv41"
I20240501 08:12:04.187471 1341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:04.217945 1341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:04.218873 1341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:04.219153 1341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:04.220150 1341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:04.221800 1341 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:04.221889 1341 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:04.222010 1341 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:04.222059 1341 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:04.254662 1341 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.6:44049
I20240501 08:12:04.254702 1467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.6:44049 every 8 connection(s)
I20240501 08:12:04.256238 1341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:12:04.266477 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 1341
I20240501 08:12:04.267102 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.7:46501
--local_ip_for_outbound_sockets=127.25.107.7
--tserver_master_addrs=127.25.107.62:43837
--webserver_port=44091
--webserver_interface=127.25.107.7
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:12:04.269539 1468 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:12:04.269944 1468 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:04.270956 1468 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:12:04.272565 32182 ts_manager.cc:194] Re-registered known tserver with Master: 37965efda2a34916b49a2357eab43bf3 (127.25.107.6:44049)
I20240501 08:12:04.273705 32182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.6:55227
I20240501 08:12:04.319939 1207 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
W20240501 08:12:04.611438 1471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:04.611786 1471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:04.611953 1471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:04.620573 1471 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:12:04.620756 1471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:04.620939 1471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.7
I20240501 08:12:04.630354 1471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.7:46501
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.107.7
--webserver_port=44091
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.7
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:04.631736 1471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:04.633625 1471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:04.646834 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:04.646754 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:04.646744 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:04.647715 1471 server_base.cc:1034] running on GCE node
I20240501 08:12:04.648335 1471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:04.649317 1471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:04.650507 1471 hybrid_clock.cc:648] HybridClock initialized: now 1714551124650438 us; error 67 us; skew 500 ppm
I20240501 08:12:04.650892 1471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:04.653209 1471 webserver.cc:460] Webserver started at http://127.25.107.7:44091/ using document root <none> and password file <none>
I20240501 08:12:04.653966 1471 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:04.654096 1471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:04.660344 1471 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 08:12:04.663771 1486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:04.665479 1471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240501 08:12:04.665669 1471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c5e88f3df73d4d4f996304b6df7e6448"
format_stamp: "Formatted at 2024-05-01 08:11:53 on dist-test-slave-dv41"
I20240501 08:12:04.666697 1471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:04.696170 1471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:04.697105 1471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:04.697396 1471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:04.698432 1471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:04.700420 1493 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 08:12:04.705915 1471 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 08:12:04.706049 1471 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240501 08:12:04.706195 1471 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 08:12:04.709318 1471 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 08:12:04.709440 1471 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240501 08:12:04.709856 1493 tablet_bootstrap.cc:492] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap starting.
I20240501 08:12:04.729831 1493 log.cc:824] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:04.734688 1493 tablet_bootstrap.cc:492] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:12:04.735502 1493 tablet_bootstrap.cc:492] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap complete.
I20240501 08:12:04.736613 1493 ts_tablet_manager.cc:1367] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.008s
I20240501 08:12:04.746860 1493 raft_consensus.cc:348] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:04.747960 1493 raft_consensus.cc:729] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e88f3df73d4d4f996304b6df7e6448, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.748704 1493 consensus_queue.cc:260] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:04.750519 1493 ts_tablet_manager.cc:1398] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20240501 08:12:04.755056 1471 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.7:46501
I20240501 08:12:04.755108 1602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.7:46501 every 8 connection(s)
I20240501 08:12:04.756493 1471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
W20240501 08:12:04.757705 1604 tablet.cc:2369] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:04.758599 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 1471
I20240501 08:12:04.758893 26028 test_util.cc:268] Using random seed: 180195659
I20240501 08:12:04.769945 1603 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:12:04.770329 1603 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:04.771386 1603 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:12:04.773756 32182 ts_manager.cc:194] Re-registered known tserver with Master: c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501)
I20240501 08:12:04.775063 32182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.7:54789
I20240501 08:12:04.777516 26028 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 08:12:04.778128 26028 test_util.cc:268] Using random seed: 180214880
I20240501 08:12:04.801680 1338 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
I20240501 08:12:04.801752 26028 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 08:12:04.806574 26028 test_util.cc:268] Using random seed: 180243305
I20240501 08:12:04.826646 26028 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 08:12:04.832502 26028 test_util.cc:268] Using random seed: 180269226
I20240501 08:12:04.893352 26028 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 08:12:04.894150 26028 test_util.cc:268] Using random seed: 180330891
I20240501 08:12:05.043855 26028 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240501 08:12:05.276859 1468 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
I20240501 08:12:05.778285 1603 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
I20240501 08:12:06.190188 1731 raft_consensus.cc:482] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:06.204233 1731 raft_consensus.cc:504] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:06.224932 1731 leader_election.cc:288] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:06.265100 32455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50cb92ecb554287977b7a980bd4d2b4" candidate_uuid: "c5e88f3df73d4d4f996304b6df7e6448" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" is_pre_election: true
I20240501 08:12:06.265385 32455 raft_consensus.cc:2454] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e88f3df73d4d4f996304b6df7e6448 in term 1.
I20240501 08:12:06.266080 1488 leader_election.cc:302] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b3e6c8e1eb974b5c9c8b7e109b214576, c5e88f3df73d4d4f996304b6df7e6448; no voters:
I20240501 08:12:06.267020 1731 raft_consensus.cc:2781] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:12:06.267196 1731 raft_consensus.cc:482] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:06.267303 1731 raft_consensus.cc:3037] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:06.275249 1731 raft_consensus.cc:504] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:06.282028 1731 leader_election.cc:288] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [CANDIDATE]: Term 2 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:06.283095 32457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50cb92ecb554287977b7a980bd4d2b4" candidate_uuid: "c5e88f3df73d4d4f996304b6df7e6448" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:12:06.283438 32457 raft_consensus.cc:3037] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:06.291425 32457 raft_consensus.cc:2454] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e88f3df73d4d4f996304b6df7e6448 in term 2.
I20240501 08:12:06.301717 1488 leader_election.cc:302] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b3e6c8e1eb974b5c9c8b7e109b214576, c5e88f3df73d4d4f996304b6df7e6448; no voters:
I20240501 08:12:06.302328 1731 raft_consensus.cc:2781] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:06.314296 1731 raft_consensus.cc:686] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 LEADER]: Becoming Leader. State: Replica: c5e88f3df73d4d4f996304b6df7e6448, State: Running, Role: LEADER
I20240501 08:12:06.324295 1731 consensus_queue.cc:237] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:06.327874 32569 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:06.346066 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:06.392745 32182 catalog_manager.cc:5484] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c5e88f3df73d4d4f996304b6df7e6448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:06.467820 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:06.538228 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:06.542735 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:06.567476 32439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:06.590855 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:06.717872 32457 raft_consensus.cc:1261] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Refusing update from remote peer c5e88f3df73d4d4f996304b6df7e6448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:12:06.718997 1731 consensus_queue.cc:1035] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.314580 32569 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:07.368381 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:07.402340 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:07.460374 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:07.465210 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:07.522931 32439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:07.550758 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:08.100010 32588 consensus_queue.cc:237] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.56, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:08.105091 32325 raft_consensus.cc:1261] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240501 08:12:08.106642 1651 consensus_queue.cc:1035] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240501 08:12:08.124644 1651 raft_consensus.cc:2932] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.57: config changed from index -1 to 57, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:08.131673 32182 catalog_manager.cc:5484] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index -1 to 57, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:08.132339 32327 raft_consensus.cc:2932] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index -1 to 57, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
W20240501 08:12:08.157120 32523 consensus_peers.cc:485] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 -> Peer 37965efda2a34916b49a2357eab43bf3 (127.25.107.6:44049): Couldn't send request to peer 37965efda2a34916b49a2357eab43bf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adbf537bdcdf4443a94799340aab4205. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:08.498458 32325 consensus_queue.cc:237] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:08.514983 32458 raft_consensus.cc:1261] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240501 08:12:08.516227 1820 consensus_queue.cc:1035] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20240501 08:12:08.523392 1698 raft_consensus.cc:2932] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:08.524603 32456 raft_consensus.cc:2932] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:08.532671 32183 catalog_manager.cc:5484] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d reported cstate change: config changed from index -1 to 66, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
W20240501 08:12:08.542759 32262 consensus_peers.cc:485] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d -> Peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5:42813): Couldn't send request to peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e794c151e6e40dbb612e64853415d9a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:08.615011 1862 ts_tablet_manager.cc:902] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: Initiating tablet copy from peer 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:12:08.761927 1862 tablet_copy_client.cc:308] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:38795
I20240501 08:12:08.803382 32609 tablet_copy_service.cc:140] P 12e44d7017614f88b3205111d85de021: Received BeginTabletCopySession request for tablet adbf537bdcdf4443a94799340aab4205 from peer 37965efda2a34916b49a2357eab43bf3 ({username='slave'} at 127.25.107.6:55123)
I20240501 08:12:08.803730 32609 tablet_copy_service.cc:161] P 12e44d7017614f88b3205111d85de021: Beginning new tablet copy session on tablet adbf537bdcdf4443a94799340aab4205 from peer 37965efda2a34916b49a2357eab43bf3 at {username='slave'} at 127.25.107.6:55123: session id = 37965efda2a34916b49a2357eab43bf3-adbf537bdcdf4443a94799340aab4205
I20240501 08:12:08.809063 32609 tablet_copy_source_session.cc:205] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:08.818588 1862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbf537bdcdf4443a94799340aab4205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.837899 1862 tablet_copy_client.cc:783] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:08.838496 1862 tablet_copy_client.cc:647] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:08.841568 1862 tablet_copy_client.cc:523] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:08.848140 1862 tablet_bootstrap.cc:492] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: Bootstrap starting.
I20240501 08:12:08.883945 1862 log.cc:824] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:08.956288 1882 ts_tablet_manager.cc:902] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Initiating tablet copy from peer 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:08.966363 32327 consensus_queue.cc:237] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:08.971078 32587 raft_consensus.cc:1261] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240501 08:12:08.977308 1566 consensus_queue.cc:1035] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240501 08:12:08.990538 1882 tablet_copy_client.cc:308] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:33223
W20240501 08:12:09.007733 32263 consensus_peers.cc:485] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d -> Peer 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729): Couldn't send request to peer 489550c4b21146918c1c9618be65b4ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4c8611ac7394bbd96cabee647f5c724. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:09.010423 1566 raft_consensus.cc:2932] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index -1 to 70, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:09.024798 32348 tablet_copy_service.cc:140] P 2219c1853e76474799f93fed12cdc12d: Received BeginTabletCopySession request for tablet 2e794c151e6e40dbb612e64853415d9a from peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e ({username='slave'} at 127.25.107.5:35565)
I20240501 08:12:09.025065 32348 tablet_copy_service.cc:161] P 2219c1853e76474799f93fed12cdc12d: Beginning new tablet copy session on tablet 2e794c151e6e40dbb612e64853415d9a from peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e at {username='slave'} at 127.25.107.5:35565: session id = 92f76e9a8b5c4bc9b49900f5ca6b4f6e-2e794c151e6e40dbb612e64853415d9a
I20240501 08:12:09.031386 32348 tablet_copy_source_session.cc:205] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:09.041458 1882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e794c151e6e40dbb612e64853415d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.056382 32182 catalog_manager.cc:5484] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d reported cstate change: config changed from index -1 to 70, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:09.059094 32589 raft_consensus.cc:2932] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index -1 to 70, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:09.092945 1882 tablet_copy_client.cc:783] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:09.094579 1882 tablet_copy_client.cc:647] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:09.099646 1882 tablet_copy_client.cc:523] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:09.107674 1882 tablet_bootstrap.cc:492] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap starting.
I20240501 08:12:09.154043 1862 tablet_bootstrap.cc:492] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=69 ignored=0} inserts{seen=754 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:09.154690 1862 tablet_bootstrap.cc:492] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: Bootstrap complete.
I20240501 08:12:09.155225 1862 ts_tablet_manager.cc:1367] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: Time spent bootstrapping tablet: real 0.307s user 0.127s sys 0.023s
I20240501 08:12:09.169618 1882 log.cc:824] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:09.170624 1862 raft_consensus.cc:348] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:09.171108 1862 raft_consensus.cc:729] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37965efda2a34916b49a2357eab43bf3, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.171993 1862 consensus_queue.cc:260] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:09.191776 1862 ts_tablet_manager.cc:1398] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3: Time spent starting tablet: real 0.036s user 0.006s sys 0.008s
I20240501 08:12:09.192839 32609 tablet_copy_service.cc:342] P 12e44d7017614f88b3205111d85de021: Request end of tablet copy session 37965efda2a34916b49a2357eab43bf3-adbf537bdcdf4443a94799340aab4205 received from {username='slave'} at 127.25.107.6:55123
I20240501 08:12:09.193022 32609 tablet_copy_service.cc:433] P 12e44d7017614f88b3205111d85de021: ending tablet copy session 37965efda2a34916b49a2357eab43bf3-adbf537bdcdf4443a94799340aab4205 on tablet adbf537bdcdf4443a94799340aab4205 with peer 37965efda2a34916b49a2357eab43bf3
I20240501 08:12:09.346361 1882 tablet_bootstrap.cc:492] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=71 ignored=0} inserts{seen=743 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:12:09.347060 1882 tablet_bootstrap.cc:492] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap complete.
I20240501 08:12:09.347620 1882 ts_tablet_manager.cc:1367] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Time spent bootstrapping tablet: real 0.242s user 0.132s sys 0.017s
I20240501 08:12:09.353945 1882 raft_consensus.cc:348] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:09.372356 1882 raft_consensus.cc:729] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f76e9a8b5c4bc9b49900f5ca6b4f6e, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.373284 1882 consensus_queue.cc:260] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:09.379877 1882 ts_tablet_manager.cc:1398] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Time spent starting tablet: real 0.032s user 0.025s sys 0.005s
I20240501 08:12:09.381500 32348 tablet_copy_service.cc:342] P 2219c1853e76474799f93fed12cdc12d: Request end of tablet copy session 92f76e9a8b5c4bc9b49900f5ca6b4f6e-2e794c151e6e40dbb612e64853415d9a received from {username='slave'} at 127.25.107.5:35565
I20240501 08:12:09.381711 32348 tablet_copy_service.cc:433] P 2219c1853e76474799f93fed12cdc12d: ending tablet copy session 92f76e9a8b5c4bc9b49900f5ca6b4f6e-2e794c151e6e40dbb612e64853415d9a on tablet 2e794c151e6e40dbb612e64853415d9a with peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e
I20240501 08:12:09.412142 1293 raft_consensus.cc:1203] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.77] Dedup: 1.72->[1.73-1.77]
I20240501 08:12:09.444636 32589 consensus_queue.cc:237] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:09.452214 32457 raft_consensus.cc:1261] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 08:12:09.458788 1008 consensus_queue.cc:1035] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.003s
I20240501 08:12:09.482903 1872 raft_consensus.cc:2932] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, VOTER c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:09.490589 32183 catalog_manager.cc:5484] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index -1 to 79, VOTER c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:09.496480 32458 raft_consensus.cc:2932] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, VOTER c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
W20240501 08:12:09.498366 32522 consensus_peers.cc:485] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 -> Peer c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501): Couldn't send request to peer c5e88f3df73d4d4f996304b6df7e6448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad5a9e14873742caa50ec838cca1c715. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:09.513733 1423 raft_consensus.cc:1203] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.56->[1.57-1.75] Dedup: 1.69->[1.70-1.75]
I20240501 08:12:09.522969 1921 ts_tablet_manager.cc:902] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: Initiating tablet copy from peer 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:09.588440 1921 tablet_copy_client.cc:308] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:33223
I20240501 08:12:09.618748 32348 tablet_copy_service.cc:140] P 2219c1853e76474799f93fed12cdc12d: Received BeginTabletCopySession request for tablet b4c8611ac7394bbd96cabee647f5c724 from peer 489550c4b21146918c1c9618be65b4ff ({username='slave'} at 127.25.107.4:43313)
I20240501 08:12:09.618980 32348 tablet_copy_service.cc:161] P 2219c1853e76474799f93fed12cdc12d: Beginning new tablet copy session on tablet b4c8611ac7394bbd96cabee647f5c724 from peer 489550c4b21146918c1c9618be65b4ff at {username='slave'} at 127.25.107.4:43313: session id = 489550c4b21146918c1c9618be65b4ff-b4c8611ac7394bbd96cabee647f5c724
I20240501 08:12:09.621959 32348 tablet_copy_source_session.cc:205] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:09.629554 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c8611ac7394bbd96cabee647f5c724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.724593 1921 tablet_copy_client.cc:783] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:09.725147 1921 tablet_copy_client.cc:647] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Starting download of 1 WAL segments...
W20240501 08:12:09.797930 32262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46875, user_credentials={real_user=slave}} blocked reactor thread for 55253.3us
I20240501 08:12:09.803329 1921 tablet_copy_client.cc:523] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:09.808656 1921 tablet_bootstrap.cc:492] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: Bootstrap starting.
I20240501 08:12:09.871104 1921 log.cc:824] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:09.987058 1950 ts_tablet_manager.cc:902] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448: Initiating tablet copy from peer 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:12:10.034584 32587 consensus_queue.cc:237] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:10.039992 32327 raft_consensus.cc:1261] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240501 08:12:10.040760 1872 consensus_queue.cc:1035] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
W20240501 08:12:10.043165 32523 consensus_peers.cc:485] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 -> Peer 37965efda2a34916b49a2357eab43bf3 (127.25.107.6:44049): Couldn't send request to peer 37965efda2a34916b49a2357eab43bf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5dbd5afc230c41509774812f082f97f0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:10.046603 1950 tablet_copy_client.cc:308] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:38795
I20240501 08:12:10.053506 1872 raft_consensus.cc:2932] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index -1 to 98, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:10.062738 32325 raft_consensus.cc:2932] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index -1 to 98, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:10.063899 32182 catalog_manager.cc:5484] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index -1 to 98, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:10.077350 32609 tablet_copy_service.cc:140] P 12e44d7017614f88b3205111d85de021: Received BeginTabletCopySession request for tablet ad5a9e14873742caa50ec838cca1c715 from peer c5e88f3df73d4d4f996304b6df7e6448 ({username='slave'} at 127.25.107.7:45921)
I20240501 08:12:10.077589 32609 tablet_copy_service.cc:161] P 12e44d7017614f88b3205111d85de021: Beginning new tablet copy session on tablet ad5a9e14873742caa50ec838cca1c715 from peer c5e88f3df73d4d4f996304b6df7e6448 at {username='slave'} at 127.25.107.7:45921: session id = c5e88f3df73d4d4f996304b6df7e6448-ad5a9e14873742caa50ec838cca1c715
I20240501 08:12:10.080475 32609 tablet_copy_source_session.cc:205] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:10.084551 1950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5a9e14873742caa50ec838cca1c715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.096604 1950 tablet_copy_client.cc:783] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:10.097121 1950 tablet_copy_client.cc:647] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:10.101452 1950 tablet_copy_client.cc:523] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:10.105927 1950 tablet_bootstrap.cc:492] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap starting.
I20240501 08:12:10.390235 1950 tablet_bootstrap.cc:492] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=89 ignored=0} inserts{seen=720 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:12:10.390928 1950 tablet_bootstrap.cc:492] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap complete.
I20240501 08:12:10.391332 1950 ts_tablet_manager.cc:1367] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448: Time spent bootstrapping tablet: real 0.286s user 0.156s sys 0.004s
I20240501 08:12:10.392490 1950 raft_consensus.cc:348] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:10.415827 1950 raft_consensus.cc:729] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e88f3df73d4d4f996304b6df7e6448, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.416311 1950 consensus_queue.cc:260] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 89, Last appended: 1.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:10.417088 1950 ts_tablet_manager.cc:1398] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448: Time spent starting tablet: real 0.026s user 0.010s sys 0.000s
I20240501 08:12:10.423535 32609 tablet_copy_service.cc:342] P 12e44d7017614f88b3205111d85de021: Request end of tablet copy session c5e88f3df73d4d4f996304b6df7e6448-ad5a9e14873742caa50ec838cca1c715 received from {username='slave'} at 127.25.107.7:45921
I20240501 08:12:10.423699 32609 tablet_copy_service.cc:433] P 12e44d7017614f88b3205111d85de021: ending tablet copy session c5e88f3df73d4d4f996304b6df7e6448-ad5a9e14873742caa50ec838cca1c715 on tablet ad5a9e14873742caa50ec838cca1c715 with peer c5e88f3df73d4d4f996304b6df7e6448
I20240501 08:12:10.444736 32587 consensus_queue.cc:237] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 1.127, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:10.465512 32457 raft_consensus.cc:1261] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20240501 08:12:10.475087 1921 tablet_bootstrap.cc:492] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=79 overwritten=0 applied=79 ignored=0} inserts{seen=710 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:10.475853 1921 tablet_bootstrap.cc:492] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: Bootstrap complete.
I20240501 08:12:10.476437 1921 ts_tablet_manager.cc:1367] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: Time spent bootstrapping tablet: real 0.668s user 0.158s sys 0.011s
I20240501 08:12:10.488049 1921 raft_consensus.cc:348] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:10.488745 1921 raft_consensus.cc:729] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489550c4b21146918c1c9618be65b4ff, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.491605 1921 consensus_queue.cc:260] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 1.79, Last appended by leader: 79, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:10.493947 1979 consensus_queue.cc:1035] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20240501 08:12:10.502146 1008 raft_consensus.cc:2932] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.128: config changed from index -1 to 128, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:10.506120 32457 raft_consensus.cc:2932] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index -1 to 128, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:10.513968 32183 catalog_manager.cc:5484] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 reported cstate change: config changed from index -1 to 128, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:10.496692 1984 ts_tablet_manager.cc:902] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3: Initiating tablet copy from peer 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
W20240501 08:12:10.525925 32524 consensus_peers.cc:485] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 -> Peer 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729): Couldn't send request to peer 489550c4b21146918c1c9618be65b4ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50c58a563cf94527895a675a1e729fe8. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:10.535439 1921 ts_tablet_manager.cc:1398] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff: Time spent starting tablet: real 0.059s user 0.023s sys 0.020s
I20240501 08:12:10.543673 32348 tablet_copy_service.cc:342] P 2219c1853e76474799f93fed12cdc12d: Request end of tablet copy session 489550c4b21146918c1c9618be65b4ff-b4c8611ac7394bbd96cabee647f5c724 received from {username='slave'} at 127.25.107.4:43313
I20240501 08:12:10.543833 32348 tablet_copy_service.cc:433] P 2219c1853e76474799f93fed12cdc12d: ending tablet copy session 489550c4b21146918c1c9618be65b4ff-b4c8611ac7394bbd96cabee647f5c724 on tablet b4c8611ac7394bbd96cabee647f5c724 with peer 489550c4b21146918c1c9618be65b4ff
I20240501 08:12:10.559358 1984 tablet_copy_client.cc:308] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:38795
I20240501 08:12:10.560236 32609 tablet_copy_service.cc:140] P 12e44d7017614f88b3205111d85de021: Received BeginTabletCopySession request for tablet 5dbd5afc230c41509774812f082f97f0 from peer 37965efda2a34916b49a2357eab43bf3 ({username='slave'} at 127.25.107.6:55123)
I20240501 08:12:10.560434 32609 tablet_copy_service.cc:161] P 12e44d7017614f88b3205111d85de021: Beginning new tablet copy session on tablet 5dbd5afc230c41509774812f082f97f0 from peer 37965efda2a34916b49a2357eab43bf3 at {username='slave'} at 127.25.107.6:55123: session id = 37965efda2a34916b49a2357eab43bf3-5dbd5afc230c41509774812f082f97f0
I20240501 08:12:10.561085 1556 raft_consensus.cc:1203] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.78->[1.79-1.95] Dedup: 1.90->[1.91-1.95]
I20240501 08:12:10.563148 32609 tablet_copy_source_session.cc:205] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:10.578500 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbd5afc230c41509774812f082f97f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.597146 1984 tablet_copy_client.cc:783] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:10.597580 1984 tablet_copy_client.cc:647] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:10.600024 1984 tablet_copy_client.cc:523] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:10.604840 1984 tablet_bootstrap.cc:492] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3: Bootstrap starting.
I20240501 08:12:10.722716 1162 raft_consensus.cc:1203] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.102] Dedup: 1.79->[1.80-1.102]
I20240501 08:12:10.929055 32587 consensus_queue.cc:237] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:10.946085 32457 raft_consensus.cc:1261] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 08:12:10.951063 1681 consensus_queue.cc:1035] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 08:12:10.964205 1008 raft_consensus.cc:2932] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index -1 to 95, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:10.970733 32182 catalog_manager.cc:5484] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index -1 to 95, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:10.971246 1984 tablet_bootstrap.cc:492] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=105 overwritten=0 applied=105 ignored=0} inserts{seen=790 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:10.971685 1984 tablet_bootstrap.cc:492] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3: Bootstrap complete.
I20240501 08:12:10.972092 1984 ts_tablet_manager.cc:1367] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3: Time spent bootstrapping tablet: real 0.367s user 0.182s sys 0.017s
I20240501 08:12:10.972373 32458 raft_consensus.cc:2932] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index -1 to 95, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:10.973138 1984 raft_consensus.cc:348] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:10.973382 1984 raft_consensus.cc:729] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37965efda2a34916b49a2357eab43bf3, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.973662 1984 consensus_queue.cc:260] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
W20240501 08:12:10.984959 32523 consensus_peers.cc:485] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 -> Peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5:42813): Couldn't send request to peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95a25d38a68a424ebf78f02d2e0750b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:10.987363 1921 ts_tablet_manager.cc:902] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: Initiating tablet copy from peer 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:12:11.011612 1984 ts_tablet_manager.cc:1398] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3: Time spent starting tablet: real 0.039s user 0.017s sys 0.020s
I20240501 08:12:11.012506 1921 tablet_copy_client.cc:308] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:38795
I20240501 08:12:11.015403 32609 tablet_copy_service.cc:342] P 12e44d7017614f88b3205111d85de021: Request end of tablet copy session 37965efda2a34916b49a2357eab43bf3-5dbd5afc230c41509774812f082f97f0 received from {username='slave'} at 127.25.107.6:55123
I20240501 08:12:11.015578 32609 tablet_copy_service.cc:433] P 12e44d7017614f88b3205111d85de021: ending tablet copy session 37965efda2a34916b49a2357eab43bf3-5dbd5afc230c41509774812f082f97f0 on tablet 5dbd5afc230c41509774812f082f97f0 with peer 37965efda2a34916b49a2357eab43bf3
I20240501 08:12:11.034833 32609 tablet_copy_service.cc:140] P 12e44d7017614f88b3205111d85de021: Received BeginTabletCopySession request for tablet 50c58a563cf94527895a675a1e729fe8 from peer 489550c4b21146918c1c9618be65b4ff ({username='slave'} at 127.25.107.4:47007)
I20240501 08:12:11.035022 32609 tablet_copy_service.cc:161] P 12e44d7017614f88b3205111d85de021: Beginning new tablet copy session on tablet 50c58a563cf94527895a675a1e729fe8 from peer 489550c4b21146918c1c9618be65b4ff at {username='slave'} at 127.25.107.4:47007: session id = 489550c4b21146918c1c9618be65b4ff-50c58a563cf94527895a675a1e729fe8
I20240501 08:12:11.037204 32609 tablet_copy_source_session.cc:205] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:11.038599 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c58a563cf94527895a675a1e729fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.046140 1921 tablet_copy_client.cc:783] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:11.046550 1921 tablet_copy_client.cc:647] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:11.049444 1921 tablet_copy_client.cc:523] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:11.053400 1921 tablet_bootstrap.cc:492] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: Bootstrap starting.
I20240501 08:12:11.248323 32589 consensus_queue.cc:237] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 1.128, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
W20240501 08:12:11.251690 32523 consensus_peers.cc:485] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 -> Peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5:42813): Couldn't send request to peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db3c2a3262974c75bba7743192d66d46. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:11.251752 32325 raft_consensus.cc:1261] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 128. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20240501 08:12:11.253198 1872 consensus_queue.cc:1035] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20240501 08:12:11.256915 1871 raft_consensus.cc:2932] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.129: config changed from index -1 to 129, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:11.262044 32325 raft_consensus.cc:2932] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index -1 to 129, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:11.262159 32182 catalog_manager.cc:5484] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index -1 to 129, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:11.407167 2054 ts_tablet_manager.cc:902] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Initiating tablet copy from peer 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:12:11.451033 2054 tablet_copy_client.cc:308] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:38795
I20240501 08:12:11.463757 1423 raft_consensus.cc:1203] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.97->[1.98-1.115] Dedup: 1.105->[1.106-1.115]
I20240501 08:12:11.474931 1921 tablet_bootstrap.cc:492] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=135 overwritten=0 applied=135 ignored=0} inserts{seen=766 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:11.475579 1921 tablet_bootstrap.cc:492] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: Bootstrap complete.
I20240501 08:12:11.475953 1921 ts_tablet_manager.cc:1367] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: Time spent bootstrapping tablet: real 0.423s user 0.195s sys 0.015s
I20240501 08:12:11.477108 1921 raft_consensus.cc:348] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:11.477473 1921 raft_consensus.cc:729] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489550c4b21146918c1c9618be65b4ff, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.477802 1921 consensus_queue.cc:260] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 135, Last appended: 1.135, Last appended by leader: 135, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:11.485508 1921 ts_tablet_manager.cc:1398] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240501 08:12:11.488397 32609 tablet_copy_service.cc:342] P 12e44d7017614f88b3205111d85de021: Request end of tablet copy session 489550c4b21146918c1c9618be65b4ff-50c58a563cf94527895a675a1e729fe8 received from {username='slave'} at 127.25.107.4:47007
I20240501 08:12:11.488574 32609 tablet_copy_service.cc:433] P 12e44d7017614f88b3205111d85de021: ending tablet copy session 489550c4b21146918c1c9618be65b4ff-50c58a563cf94527895a675a1e729fe8 on tablet 50c58a563cf94527895a675a1e729fe8 with peer 489550c4b21146918c1c9618be65b4ff
I20240501 08:12:11.500312 32609 tablet_copy_service.cc:140] P 12e44d7017614f88b3205111d85de021: Received BeginTabletCopySession request for tablet 95a25d38a68a424ebf78f02d2e0750b1 from peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e ({username='slave'} at 127.25.107.5:56711)
I20240501 08:12:11.500505 32609 tablet_copy_service.cc:161] P 12e44d7017614f88b3205111d85de021: Beginning new tablet copy session on tablet 95a25d38a68a424ebf78f02d2e0750b1 from peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e at {username='slave'} at 127.25.107.5:56711: session id = 92f76e9a8b5c4bc9b49900f5ca6b4f6e-95a25d38a68a424ebf78f02d2e0750b1
I20240501 08:12:11.503582 32609 tablet_copy_source_session.cc:205] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:11.505908 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a25d38a68a424ebf78f02d2e0750b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.512977 2054 tablet_copy_client.cc:783] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:11.513279 2054 tablet_copy_client.cc:647] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:11.515594 2054 tablet_copy_client.cc:523] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:11.519973 2054 tablet_bootstrap.cc:492] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap starting.
I20240501 08:12:11.914552 2072 ts_tablet_manager.cc:902] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Initiating tablet copy from peer 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:12:11.920295 1162 raft_consensus.cc:1203] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.127->[1.128-1.152] Dedup: 1.135->[1.136-1.152]
I20240501 08:12:12.143029 2072 tablet_copy_client.cc:308] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:38795
I20240501 08:12:12.162134 32609 tablet_copy_service.cc:140] P 12e44d7017614f88b3205111d85de021: Received BeginTabletCopySession request for tablet db3c2a3262974c75bba7743192d66d46 from peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e ({username='slave'} at 127.25.107.5:56711)
I20240501 08:12:12.162385 32609 tablet_copy_service.cc:161] P 12e44d7017614f88b3205111d85de021: Beginning new tablet copy session on tablet db3c2a3262974c75bba7743192d66d46 from peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e at {username='slave'} at 127.25.107.5:56711: session id = 92f76e9a8b5c4bc9b49900f5ca6b4f6e-db3c2a3262974c75bba7743192d66d46
I20240501 08:12:12.165319 32609 tablet_copy_source_session.cc:205] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:12.167153 2072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3c2a3262974c75bba7743192d66d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.193992 2072 tablet_copy_client.cc:783] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:12.194450 2072 tablet_copy_client.cc:647] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:12.221305 2072 tablet_copy_client.cc:523] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:12.224843 2072 tablet_bootstrap.cc:492] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap starting.
I20240501 08:12:12.372799 32587 consensus_queue.cc:237] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.126, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:12.388118 32455 raft_consensus.cc:1261] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240501 08:12:12.390064 1008 consensus_queue.cc:1035] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
W20240501 08:12:12.396101 32523 consensus_peers.cc:485] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 -> Peer 37965efda2a34916b49a2357eab43bf3 (127.25.107.6:44049): Couldn't send request to peer 37965efda2a34916b49a2357eab43bf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bce7a736006455aa01f06e5542022ec. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:12.405398 1008 raft_consensus.cc:2932] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.127: config changed from index -1 to 127, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:12.410300 32458 raft_consensus.cc:2932] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index -1 to 127, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:12.416767 32182 catalog_manager.cc:5484] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index -1 to 127, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:12.766463 2072 tablet_bootstrap.cc:492] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=137 overwritten=0 applied=137 ignored=0} inserts{seen=822 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:12.767068 2072 tablet_bootstrap.cc:492] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap complete.
I20240501 08:12:12.770802 2072 ts_tablet_manager.cc:1367] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Time spent bootstrapping tablet: real 0.546s user 0.201s sys 0.020s
I20240501 08:12:12.771106 2054 tablet_bootstrap.cc:492] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=103 overwritten=0 applied=103 ignored=0} inserts{seen=788 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:12.775544 2054 tablet_bootstrap.cc:492] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap complete.
I20240501 08:12:12.776486 2054 ts_tablet_manager.cc:1367] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Time spent bootstrapping tablet: real 1.257s user 0.174s sys 0.032s
I20240501 08:12:12.777192 2072 raft_consensus.cc:348] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:12.782060 2072 raft_consensus.cc:729] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f76e9a8b5c4bc9b49900f5ca6b4f6e, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.782522 2072 consensus_queue.cc:260] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 137, Last appended: 1.137, Last appended by leader: 137, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:12.784227 2054 raft_consensus.cc:348] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:12.785228 2054 raft_consensus.cc:729] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f76e9a8b5c4bc9b49900f5ca6b4f6e, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.785755 2054 consensus_queue.cc:260] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 103, Last appended: 1.103, Last appended by leader: 103, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:12.798857 2054 ts_tablet_manager.cc:1398] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Time spent starting tablet: real 0.017s user 0.000s sys 0.005s
I20240501 08:12:12.815236 32609 tablet_copy_service.cc:342] P 12e44d7017614f88b3205111d85de021: Request end of tablet copy session 92f76e9a8b5c4bc9b49900f5ca6b4f6e-95a25d38a68a424ebf78f02d2e0750b1 received from {username='slave'} at 127.25.107.5:56711
I20240501 08:12:12.815423 32609 tablet_copy_service.cc:433] P 12e44d7017614f88b3205111d85de021: ending tablet copy session 92f76e9a8b5c4bc9b49900f5ca6b4f6e-95a25d38a68a424ebf78f02d2e0750b1 on tablet 95a25d38a68a424ebf78f02d2e0750b1 with peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e
I20240501 08:12:12.907311 2072 ts_tablet_manager.cc:1398] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Time spent starting tablet: real 0.136s user 0.034s sys 0.042s
I20240501 08:12:12.910516 32609 tablet_copy_service.cc:342] P 12e44d7017614f88b3205111d85de021: Request end of tablet copy session 92f76e9a8b5c4bc9b49900f5ca6b4f6e-db3c2a3262974c75bba7743192d66d46 received from {username='slave'} at 127.25.107.5:56711
I20240501 08:12:12.910704 32609 tablet_copy_service.cc:433] P 12e44d7017614f88b3205111d85de021: ending tablet copy session 92f76e9a8b5c4bc9b49900f5ca6b4f6e-db3c2a3262974c75bba7743192d66d46 on tablet db3c2a3262974c75bba7743192d66d46 with peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e
I20240501 08:12:12.939062 2122 ts_tablet_manager.cc:902] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3: Initiating tablet copy from peer 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:12:13.056586 1293 raft_consensus.cc:1203] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.94->[1.95-1.119] Dedup: 1.103->[1.104-1.119]
I20240501 08:12:13.092949 32325 consensus_queue.cc:237] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 154, Committed index: 154, Last appended: 1.155, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:13.115175 32589 raft_consensus.cc:1261] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
W20240501 08:12:13.116966 32263 consensus_peers.cc:485] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d -> Peer 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729): Couldn't send request to peer 489550c4b21146918c1c9618be65b4ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4ec90b4ddce408a81f588fa7546e0dd. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:13.117395 1566 consensus_queue.cc:1035] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 154, Time since last communication: 0.000s
I20240501 08:12:13.130750 2122 tablet_copy_client.cc:308] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:38795
I20240501 08:12:13.132848 32609 tablet_copy_service.cc:140] P 12e44d7017614f88b3205111d85de021: Received BeginTabletCopySession request for tablet 8bce7a736006455aa01f06e5542022ec from peer 37965efda2a34916b49a2357eab43bf3 ({username='slave'} at 127.25.107.6:55123)
I20240501 08:12:13.133033 32609 tablet_copy_service.cc:161] P 12e44d7017614f88b3205111d85de021: Beginning new tablet copy session on tablet 8bce7a736006455aa01f06e5542022ec from peer 37965efda2a34916b49a2357eab43bf3 at {username='slave'} at 127.25.107.6:55123: session id = 37965efda2a34916b49a2357eab43bf3-8bce7a736006455aa01f06e5542022ec
I20240501 08:12:13.136050 32609 tablet_copy_source_session.cc:205] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:13.141005 2122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bce7a736006455aa01f06e5542022ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.147281 2122 tablet_copy_client.cc:783] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:13.147650 2122 tablet_copy_client.cc:647] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:13.150630 2092 raft_consensus.cc:2932] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.156: config changed from index -1 to 156, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:13.153210 32589 raft_consensus.cc:2932] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index -1 to 156, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:13.161643 32181 catalog_manager.cc:5484] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d reported cstate change: config changed from index -1 to 156, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:13.179677 2122 tablet_copy_client.cc:523] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:13.183369 2122 tablet_bootstrap.cc:492] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3: Bootstrap starting.
I20240501 08:12:13.379083 1293 raft_consensus.cc:1203] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.128->[1.129-1.155] Dedup: 1.137->[1.138-1.155]
I20240501 08:12:13.620833 1556 consensus_queue.cc:237] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:13.654923 32456 raft_consensus.cc:1261] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Refusing update from remote peer c5e88f3df73d4d4f996304b6df7e6448: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:12:13.658568 2118 consensus_queue.cc:1035] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:13.671265 2122 tablet_bootstrap.cc:492] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=137 overwritten=0 applied=137 ignored=0} inserts{seen=822 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:13.671924 2122 tablet_bootstrap.cc:492] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3: Bootstrap complete.
I20240501 08:12:13.672307 2122 ts_tablet_manager.cc:1367] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3: Time spent bootstrapping tablet: real 0.489s user 0.205s sys 0.014s
I20240501 08:12:13.673202 2122 raft_consensus.cc:348] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:13.673545 2122 raft_consensus.cc:729] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37965efda2a34916b49a2357eab43bf3, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.673933 2122 consensus_queue.cc:260] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 137, Last appended: 1.137, Last appended by leader: 137, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:13.721293 2174 ts_tablet_manager.cc:902] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff: Initiating tablet copy from peer 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:13.736492 2173 raft_consensus.cc:2932] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:13.763140 2122 ts_tablet_manager.cc:1398] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3: Time spent starting tablet: real 0.091s user 0.012s sys 0.026s
I20240501 08:12:13.764207 32609 tablet_copy_service.cc:342] P 12e44d7017614f88b3205111d85de021: Request end of tablet copy session 37965efda2a34916b49a2357eab43bf3-8bce7a736006455aa01f06e5542022ec received from {username='slave'} at 127.25.107.6:55123
I20240501 08:12:13.764384 32609 tablet_copy_service.cc:433] P 12e44d7017614f88b3205111d85de021: ending tablet copy session 37965efda2a34916b49a2357eab43bf3-8bce7a736006455aa01f06e5542022ec on tablet 8bce7a736006455aa01f06e5542022ec with peer 37965efda2a34916b49a2357eab43bf3
I20240501 08:12:13.769104 32182 catalog_manager.cc:5484] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 reported cstate change: config changed from index -1 to 3, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New cstate: current_term: 2 leader_uuid: "c5e88f3df73d4d4f996304b6df7e6448" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:13.769984 32456 raft_consensus.cc:2932] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:13.783864 2174 tablet_copy_client.cc:308] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:33223
I20240501 08:12:13.787752 32348 tablet_copy_service.cc:140] P 2219c1853e76474799f93fed12cdc12d: Received BeginTabletCopySession request for tablet f4ec90b4ddce408a81f588fa7546e0dd from peer 489550c4b21146918c1c9618be65b4ff ({username='slave'} at 127.25.107.4:43313)
I20240501 08:12:13.787981 32348 tablet_copy_service.cc:161] P 2219c1853e76474799f93fed12cdc12d: Beginning new tablet copy session on tablet f4ec90b4ddce408a81f588fa7546e0dd from peer 489550c4b21146918c1c9618be65b4ff at {username='slave'} at 127.25.107.4:43313: session id = 489550c4b21146918c1c9618be65b4ff-f4ec90b4ddce408a81f588fa7546e0dd
I20240501 08:12:13.792218 32348 tablet_copy_source_session.cc:205] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:13.796147 2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ec90b4ddce408a81f588fa7546e0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.828361 2174 tablet_copy_client.cc:783] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:13.828980 2174 tablet_copy_client.cc:647] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:13.836426 2174 tablet_copy_client.cc:523] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 08:12:13.866652 1489 consensus_peers.cc:485] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 -> Peer 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729): Couldn't send request to peer 489550c4b21146918c1c9618be65b4ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a50cb92ecb554287977b7a980bd4d2b4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:13.868928 2174 tablet_bootstrap.cc:492] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff: Bootstrap starting.
I20240501 08:12:13.998208 1423 raft_consensus.cc:1203] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.126->[1.127-1.147] Dedup: 1.137->[1.138-1.147]
I20240501 08:12:14.060319 2194 ts_tablet_manager.cc:902] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff: Initiating tablet copy from peer c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501)
I20240501 08:12:14.101279 2194 tablet_copy_client.cc:308] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.7:46501
I20240501 08:12:14.152081 1577 tablet_copy_service.cc:140] P c5e88f3df73d4d4f996304b6df7e6448: Received BeginTabletCopySession request for tablet a50cb92ecb554287977b7a980bd4d2b4 from peer 489550c4b21146918c1c9618be65b4ff ({username='slave'} at 127.25.107.4:52815)
I20240501 08:12:14.152423 1577 tablet_copy_service.cc:161] P c5e88f3df73d4d4f996304b6df7e6448: Beginning new tablet copy session on tablet a50cb92ecb554287977b7a980bd4d2b4 from peer 489550c4b21146918c1c9618be65b4ff at {username='slave'} at 127.25.107.4:52815: session id = 489550c4b21146918c1c9618be65b4ff-a50cb92ecb554287977b7a980bd4d2b4
I20240501 08:12:14.156263 1577 tablet_copy_source_session.cc:205] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:14.158164 2194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50cb92ecb554287977b7a980bd4d2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.175897 2194 tablet_copy_client.cc:783] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:14.178326 2194 tablet_copy_client.cc:647] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:14.180444 2194 tablet_copy_client.cc:523] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:14.191488 2194 tablet_bootstrap.cc:492] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff: Bootstrap starting.
I20240501 08:12:14.217660 2194 tablet_bootstrap.cc:492] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:14.218183 2194 tablet_bootstrap.cc:492] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff: Bootstrap complete.
I20240501 08:12:14.218502 2194 ts_tablet_manager.cc:1367] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20240501 08:12:14.219331 2194 raft_consensus.cc:348] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:14.219560 2194 raft_consensus.cc:729] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489550c4b21146918c1c9618be65b4ff, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.219913 2194 consensus_queue.cc:260] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:14.232689 2194 ts_tablet_manager.cc:1398] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240501 08:12:14.235390 1577 tablet_copy_service.cc:342] P c5e88f3df73d4d4f996304b6df7e6448: Request end of tablet copy session 489550c4b21146918c1c9618be65b4ff-a50cb92ecb554287977b7a980bd4d2b4 received from {username='slave'} at 127.25.107.4:52815
I20240501 08:12:14.235622 1577 tablet_copy_service.cc:433] P c5e88f3df73d4d4f996304b6df7e6448: ending tablet copy session 489550c4b21146918c1c9618be65b4ff-a50cb92ecb554287977b7a980bd4d2b4 on tablet a50cb92ecb554287977b7a980bd4d2b4 with peer 489550c4b21146918c1c9618be65b4ff
I20240501 08:12:14.290663 2174 tablet_bootstrap.cc:492] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=159 overwritten=0 applied=159 ignored=0} inserts{seen=790 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:14.291319 2174 tablet_bootstrap.cc:492] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff: Bootstrap complete.
I20240501 08:12:14.291759 2174 ts_tablet_manager.cc:1367] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff: Time spent bootstrapping tablet: real 0.438s user 0.224s sys 0.032s
I20240501 08:12:14.292762 2174 raft_consensus.cc:348] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:14.293066 2174 raft_consensus.cc:729] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489550c4b21146918c1c9618be65b4ff, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.293387 2174 consensus_queue.cc:260] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 159, Last appended: 1.159, Last appended by leader: 159, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:14.294059 2174 ts_tablet_manager.cc:1398] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:14.300436 32348 tablet_copy_service.cc:342] P 2219c1853e76474799f93fed12cdc12d: Request end of tablet copy session 489550c4b21146918c1c9618be65b4ff-f4ec90b4ddce408a81f588fa7546e0dd received from {username='slave'} at 127.25.107.4:43313
I20240501 08:12:14.300634 32348 tablet_copy_service.cc:433] P 2219c1853e76474799f93fed12cdc12d: ending tablet copy session 489550c4b21146918c1c9618be65b4ff-f4ec90b4ddce408a81f588fa7546e0dd on tablet f4ec90b4ddce408a81f588fa7546e0dd with peer 489550c4b21146918c1c9618be65b4ff
I20240501 08:12:14.340219 1162 raft_consensus.cc:1203] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.155->[1.156-1.165] Dedup: 1.159->[1.160-1.165]
I20240501 08:12:14.459475 32326 consensus_queue.cc:237] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 1.155, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:14.462575 32589 raft_consensus.cc:1261] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240501 08:12:14.464221 2094 consensus_queue.cc:1035] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.001s
I20240501 08:12:14.467633 1566 raft_consensus.cc:2932] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.156: config changed from index -1 to 156, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:14.468390 32589 raft_consensus.cc:2932] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index -1 to 156, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:14.477523 32183 catalog_manager.cc:5484] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index -1 to 156, VOTER 37965efda2a34916b49a2357eab43bf3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
W20240501 08:12:14.527770 32262 consensus_peers.cc:485] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d -> Peer 37965efda2a34916b49a2357eab43bf3 (127.25.107.6:44049): Couldn't send request to peer 37965efda2a34916b49a2357eab43bf3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 841db634d09c4ef0bdfda733a489af5d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:14.610137 1160 raft_consensus.cc:1203] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
W20240501 08:12:14.684199 32262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:42813, user_credentials={real_user=slave}} blocked reactor thread for 45387.3us
I20240501 08:12:15.092744 2245 ts_tablet_manager.cc:902] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3: Initiating tablet copy from peer 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:15.199368 2245 tablet_copy_client.cc:308] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:33223
I20240501 08:12:15.258596 32348 tablet_copy_service.cc:140] P 2219c1853e76474799f93fed12cdc12d: Received BeginTabletCopySession request for tablet 841db634d09c4ef0bdfda733a489af5d from peer 37965efda2a34916b49a2357eab43bf3 ({username='slave'} at 127.25.107.6:46675)
I20240501 08:12:15.258819 32348 tablet_copy_service.cc:161] P 2219c1853e76474799f93fed12cdc12d: Beginning new tablet copy session on tablet 841db634d09c4ef0bdfda733a489af5d from peer 37965efda2a34916b49a2357eab43bf3 at {username='slave'} at 127.25.107.6:46675: session id = 37965efda2a34916b49a2357eab43bf3-841db634d09c4ef0bdfda733a489af5d
I20240501 08:12:15.261898 32348 tablet_copy_source_session.cc:205] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:15.264284 2245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 841db634d09c4ef0bdfda733a489af5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.276558 2245 tablet_copy_client.cc:783] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:15.277911 2245 tablet_copy_client.cc:647] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:15.284441 2245 tablet_copy_client.cc:523] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:15.295989 2245 tablet_bootstrap.cc:492] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3: Bootstrap starting.
I20240501 08:12:15.922246 2245 tablet_bootstrap.cc:492] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=167 overwritten=0 applied=167 ignored=0} inserts{seen=798 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:15.931097 2245 tablet_bootstrap.cc:492] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3: Bootstrap complete.
I20240501 08:12:15.931456 2245 ts_tablet_manager.cc:1367] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3: Time spent bootstrapping tablet: real 0.636s user 0.244s sys 0.023s
I20240501 08:12:15.932291 2245 raft_consensus.cc:348] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:15.932526 2245 raft_consensus.cc:729] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37965efda2a34916b49a2357eab43bf3, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.932773 2245 consensus_queue.cc:260] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 167, Last appended: 1.167, Last appended by leader: 167, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
W20240501 08:12:16.048301 32262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46875, user_credentials={real_user=slave}} blocked reactor thread for 47631us
I20240501 08:12:16.212445 2245 ts_tablet_manager.cc:1398] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3: Time spent starting tablet: real 0.281s user 0.096s sys 0.091s
I20240501 08:12:16.218053 32348 tablet_copy_service.cc:342] P 2219c1853e76474799f93fed12cdc12d: Request end of tablet copy session 37965efda2a34916b49a2357eab43bf3-841db634d09c4ef0bdfda733a489af5d received from {username='slave'} at 127.25.107.6:46675
I20240501 08:12:16.218214 32348 tablet_copy_service.cc:433] P 2219c1853e76474799f93fed12cdc12d: ending tablet copy session 37965efda2a34916b49a2357eab43bf3-841db634d09c4ef0bdfda733a489af5d on tablet 841db634d09c4ef0bdfda733a489af5d with peer 37965efda2a34916b49a2357eab43bf3
I20240501 08:12:16.458761 1421 raft_consensus.cc:1203] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.155->[1.156-1.181] Dedup: 1.167->[1.168-1.181]
I20240501 08:12:16.556172 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:16.564455 32569 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:16.607893 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:16.638643 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:16.755070 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:16.772712 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:16.797341 32439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:17.771734 32327 consensus_queue.cc:237] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:17.774644 1421 raft_consensus.cc:1261] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 1 index: 206. (index mismatch)
I20240501 08:12:17.775595 2279 consensus_queue.cc:1035] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20240501 08:12:17.780164 1698 raft_consensus.cc:2932] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.206: config changed from index 156 to 206, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:17.781173 1421 raft_consensus.cc:2932] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Committing config change with OpId 1.206: config changed from index 156 to 206, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:17.795852 32182 catalog_manager.cc:5484] T 841db634d09c4ef0bdfda733a489af5d P 2219c1853e76474799f93fed12cdc12d reported cstate change: config changed from index 156 to 206, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:17.894140 32569 tablet_service.cc:1514] Processing DeleteTablet for tablet 841db634d09c4ef0bdfda733a489af5d with delete_type TABLET_DATA_TOMBSTONED (TS 12e44d7017614f88b3205111d85de021 not found in new config with opid_index 206) from {username='slave'} at 127.0.0.1:39982
I20240501 08:12:17.897109 2348 tablet_replica.cc:330] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: stopping tablet replica
I20240501 08:12:17.897670 2348 raft_consensus.cc:2229] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.906126 2348 raft_consensus.cc:2258] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.908988 2348 ts_tablet_manager.cc:1875] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:17.928355 2348 ts_tablet_manager.cc:1888] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20240501 08:12:17.928893 2348 log.cc:1196] T 841db634d09c4ef0bdfda733a489af5d P 12e44d7017614f88b3205111d85de021: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/841db634d09c4ef0bdfda733a489af5d
I20240501 08:12:17.954617 32173 catalog_manager.cc:4830] TS 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795): tablet 841db634d09c4ef0bdfda733a489af5d (table rebalance_test_table_0 [id=56218deb6dfb40f5896115db6d37a8ec]) successfully deleted
I20240501 08:12:19.595999 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:19.608582 32569 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:19.688571 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:19.695950 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:19.734128 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:19.821368 32439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:19.836447 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:20.538172 1556 consensus_queue.cc:237] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:20.593787 1160 raft_consensus.cc:1261] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Refusing update from remote peer c5e88f3df73d4d4f996304b6df7e6448: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:20.595604 2362 consensus_queue.cc:1035] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:20.614111 2439 raft_consensus.cc:2932] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:20.614917 1160 raft_consensus.cc:2932] T a50cb92ecb554287977b7a980bd4d2b4 P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:20.629500 32182 catalog_manager.cc:5484] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 reported cstate change: config changed from index 3 to 4, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New cstate: current_term: 2 leader_uuid: "c5e88f3df73d4d4f996304b6df7e6448" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:20.690977 32439 tablet_service.cc:1514] Processing DeleteTablet for tablet a50cb92ecb554287977b7a980bd4d2b4 with delete_type TABLET_DATA_TOMBSTONED (TS b3e6c8e1eb974b5c9c8b7e109b214576 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55378
I20240501 08:12:20.708815 2451 tablet_replica.cc:330] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: stopping tablet replica
I20240501 08:12:20.709267 2451 raft_consensus.cc:2229] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:20.709584 2451 raft_consensus.cc:2258] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:20.713433 2451 ts_tablet_manager.cc:1875] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:20.727382 2451 ts_tablet_manager.cc:1888] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 08:12:20.728020 2451 log.cc:1196] T a50cb92ecb554287977b7a980bd4d2b4 P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/a50cb92ecb554287977b7a980bd4d2b4
I20240501 08:12:20.729038 32172 catalog_manager.cc:4830] TS b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875): tablet a50cb92ecb554287977b7a980bd4d2b4 (table TestTable [id=befd81a3b75d41868559acd068af1d16]) successfully deleted
I20240501 08:12:21.888422 32569 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:21.964334 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:22.080765 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:12:22.096457 32263 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.3:38795, user_credentials={real_user=slave}} blocked reactor thread for 60903.1us
I20240501 08:12:22.122685 32439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:22.130096 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:22.145524 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:12:22.234555 32263 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.3:38795, user_credentials={real_user=slave}} blocked reactor thread for 125185us
I20240501 08:12:22.397492 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:12:22.481492 32523 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46875, user_credentials={real_user=slave}} blocked reactor thread for 52255.6us
I20240501 08:12:23.018836 32325 consensus_queue.cc:237] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 1.292, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:23.025842 1162 raft_consensus.cc:1261] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20240501 08:12:23.029265 2436 consensus_queue.cc:1035] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 291, Time since last communication: 0.000s
I20240501 08:12:23.033329 2518 raft_consensus.cc:2932] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.293: config changed from index 156 to 293, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:23.034489 1162 raft_consensus.cc:2932] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 156 to 293, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:23.044430 32182 catalog_manager.cc:5484] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d reported cstate change: config changed from index 156 to 293, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:23.192139 32569 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ec90b4ddce408a81f588fa7546e0dd with delete_type TABLET_DATA_TOMBSTONED (TS 12e44d7017614f88b3205111d85de021 not found in new config with opid_index 293) from {username='slave'} at 127.0.0.1:39982
I20240501 08:12:23.267338 2548 tablet_replica.cc:330] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021: stopping tablet replica
I20240501 08:12:23.269682 2548 raft_consensus.cc:2229] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:23.270505 2548 raft_consensus.cc:2258] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:23.272708 2548 ts_tablet_manager.cc:1875] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:23.361512 2548 ts_tablet_manager.cc:1888] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.292
I20240501 08:12:23.362622 2548 log.cc:1196] T f4ec90b4ddce408a81f588fa7546e0dd P 12e44d7017614f88b3205111d85de021: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4ec90b4ddce408a81f588fa7546e0dd
I20240501 08:12:23.363606 32173 catalog_manager.cc:4830] TS 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795): tablet f4ec90b4ddce408a81f588fa7546e0dd (table rebalance_test_table_2 [id=821c266bb4a949cf878e5f7a6df80abf]) successfully deleted
I20240501 08:12:23.917224 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:24.029829 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:24.047293 32569 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:24.064360 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:24.170966 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:24.211094 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:24.293761 32439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:12:24.521184 32262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:42813, user_credentials={real_user=slave}} blocked reactor thread for 45517.4us
I20240501 08:12:24.965257 32588 consensus_queue.cc:237] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 1.284, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:24.973536 1423 raft_consensus.cc:1261] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 285. (index mismatch)
I20240501 08:12:24.974584 2368 consensus_queue.cc:1035] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 283, Time since last communication: 0.001s
I20240501 08:12:24.980190 2589 raft_consensus.cc:2932] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.285: config changed from index 127 to 285, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:24.985010 1420 raft_consensus.cc:2932] T 8bce7a736006455aa01f06e5542022ec P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Committing config change with OpId 1.285: config changed from index 127 to 285, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:24.987063 32182 catalog_manager.cc:5484] T 8bce7a736006455aa01f06e5542022ec P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index 127 to 285, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:25.072489 32439 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bce7a736006455aa01f06e5542022ec with delete_type TABLET_DATA_TOMBSTONED (TS b3e6c8e1eb974b5c9c8b7e109b214576 not found in new config with opid_index 285) from {username='slave'} at 127.0.0.1:55378
I20240501 08:12:25.073328 2636 tablet_replica.cc:330] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: stopping tablet replica
I20240501 08:12:25.073971 2636 raft_consensus.cc:2229] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:25.074504 2636 pending_rounds.cc:62] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: Trying to abort 1 pending ops.
I20240501 08:12:25.074690 2636 pending_rounds.cc:68] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: Aborting op as it isn't in flight: id { term: 1 index: 284 } timestamp: 7022801489752850432 op_type: WRITE_OP write_request { tablet_id: "8bce7a736006455aa01f06e5542022ec" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\007\n\312 \271(\243z\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""4acd7340ec932a3b" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022801489623425024 authz_token { token_data: "\010\200\365\307\261\006\"3\n\005slave\022*\n 3910e4f153974479aa62dc08cd2c1ac3\020\001\030\001 \001(\001" signature: "<redacted>""w\216h\265\277\255SN%\277\002t-\356b\273dt\376\217\205\374\251\373\255B[\016\226\267\224\376\225FM$\224\225K\371\311\302|B7\231(\302\331cT\222\304\0020\244]\365\"nw\215i\310" signing_key_seq_num: 0 } } request_id { client_id: "a6e17c91c7954c49a82255a80db1e7cf" seq_no: 829 first_incomplete_seq_no: 829 attempt_no: 0 }
I20240501 08:12:25.075830 2636 raft_consensus.cc:2258] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.077116 2636 ts_tablet_manager.cc:1875] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:25.094362 2636 ts_tablet_manager.cc:1888] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.284
I20240501 08:12:25.094616 2636 log.cc:1196] T 8bce7a736006455aa01f06e5542022ec P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/8bce7a736006455aa01f06e5542022ec
I20240501 08:12:25.097316 32172 catalog_manager.cc:4830] TS b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875): tablet 8bce7a736006455aa01f06e5542022ec (table rebalance_test_table_1 [id=3910e4f153974479aa62dc08cd2c1ac3]) successfully deleted
I20240501 08:12:26.566237 32569 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:26.576745 32439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:26.576764 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:26.587277 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:26.608585 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:26.653748 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:26.683591 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:27.385761 32587 consensus_queue.cc:237] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 1.337, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:27.389132 1292 raft_consensus.cc:1261] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 337. Preceding OpId from leader: term: 1 index: 338. (index mismatch)
I20240501 08:12:27.390867 2368 consensus_queue.cc:1035] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I20240501 08:12:27.409970 2368 raft_consensus.cc:2932] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.338: config changed from index 129 to 338, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:27.421723 1292 raft_consensus.cc:2932] T db3c2a3262974c75bba7743192d66d46 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Committing config change with OpId 1.338: config changed from index 129 to 338, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:27.422362 32182 catalog_manager.cc:5484] T db3c2a3262974c75bba7743192d66d46 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index 129 to 338, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:27.541070 32306 tablet_service.cc:1514] Processing DeleteTablet for tablet db3c2a3262974c75bba7743192d66d46 with delete_type TABLET_DATA_TOMBSTONED (TS 2219c1853e76474799f93fed12cdc12d not found in new config with opid_index 338) from {username='slave'} at 127.0.0.1:60500
I20240501 08:12:27.564605 2733 tablet_replica.cc:330] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d: stopping tablet replica
I20240501 08:12:27.567986 2733 raft_consensus.cc:2229] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:27.569342 2733 raft_consensus.cc:2258] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:27.573096 2733 ts_tablet_manager.cc:1875] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:27.579291 2733 ts_tablet_manager.cc:1888] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.337
I20240501 08:12:27.579587 2733 log.cc:1196] T db3c2a3262974c75bba7743192d66d46 P 2219c1853e76474799f93fed12cdc12d: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/db3c2a3262974c75bba7743192d66d46
I20240501 08:12:27.581882 32171 catalog_manager.cc:4830] TS 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223): tablet db3c2a3262974c75bba7743192d66d46 (table rebalance_test_table_0 [id=56218deb6dfb40f5896115db6d37a8ec]) successfully deleted
I20240501 08:12:28.550088 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:28.565655 32569 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:28.585489 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:28.588161 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:28.602316 32439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:28.660859 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:28.662595 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:12:28.785320 32262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46875, user_credentials={real_user=slave}} blocked reactor thread for 45906.5us
W20240501 08:12:28.842023 32262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:42813, user_credentials={real_user=slave}} blocked reactor thread for 50122.6us
I20240501 08:12:28.988230 32588 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "95a25d38a68a424ebf78f02d2e0750b1"
dest_uuid: "12e44d7017614f88b3205111d85de021"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51696
I20240501 08:12:28.988550 32588 raft_consensus.cc:595] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Received request to transfer leadership
W20240501 08:12:29.021323 32547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
W20240501 08:12:29.032541 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
W20240501 08:12:29.045796 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
W20240501 08:12:29.074373 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
W20240501 08:12:29.090742 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
W20240501 08:12:29.103078 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
W20240501 08:12:29.116127 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
W20240501 08:12:29.128813 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
W20240501 08:12:29.141031 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
W20240501 08:12:29.153882 32547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
W20240501 08:12:29.168982 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
I20240501 08:12:29.171108 2384 raft_consensus.cc:979] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: : Instructing follower b3e6c8e1eb974b5c9c8b7e109b214576 to start an election
I20240501 08:12:29.171550 2384 raft_consensus.cc:1067] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Signalling peer b3e6c8e1eb974b5c9c8b7e109b214576 to start an election
I20240501 08:12:29.172644 32456 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "95a25d38a68a424ebf78f02d2e0750b1"
dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
from {username='slave'} at 127.25.107.3:41173
I20240501 08:12:29.172956 32456 raft_consensus.cc:482] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:29.173079 32456 raft_consensus.cc:3037] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:29.177017 32456 raft_consensus.cc:504] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:29.182961 32456 leader_election.cc:288] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 election: Requested vote from peers 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795), 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5:42813)
W20240501 08:12:29.191696 32547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Service unavailable: leader transfer in progress
I20240501 08:12:29.194974 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a25d38a68a424ebf78f02d2e0750b1" candidate_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" candidate_term: 2 candidate_status { last_received { term: 1 index: 399 } } ignore_live_leader: true dest_uuid: "12e44d7017614f88b3205111d85de021"
I20240501 08:12:29.195317 32587 raft_consensus.cc:3032] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:29.195451 32587 raft_consensus.cc:729] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Running, Role: LEADER
I20240501 08:12:29.195858 32587 consensus_queue.cc:260] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 399, Committed index: 399, Last appended: 1.399, Last appended by leader: 399, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:29.196389 32587 raft_consensus.cc:3037] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:12:29.207293 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57658: Illegal state: replica 12e44d7017614f88b3205111d85de021 is not leader of this config: current role FOLLOWER
I20240501 08:12:29.218855 32587 raft_consensus.cc:2454] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e6c8e1eb974b5c9c8b7e109b214576 in term 2.
I20240501 08:12:29.220335 32394 leader_election.cc:302] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:29.220799 1906 raft_consensus.cc:2781] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:29.220955 1906 raft_consensus.cc:686] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Becoming Leader. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Running, Role: LEADER
I20240501 08:12:29.221305 1906 consensus_queue.cc:237] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 399, Committed index: 399, Last appended: 1.399, Last appended by leader: 399, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:29.227583 32183 catalog_manager.cc:5484] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 reported cstate change: term changed from 1 to 2, leader changed from 12e44d7017614f88b3205111d85de021 (127.25.107.3) to b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2). New cstate: current_term: 2 leader_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:29.254190 32587 raft_consensus.cc:1261] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 1 index: 399. Preceding OpId from leader: term: 2 index: 401. (index mismatch)
I20240501 08:12:29.255506 1859 consensus_queue.cc:1035] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
I20240501 08:12:29.287700 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a25d38a68a424ebf78f02d2e0750b1" candidate_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" candidate_term: 2 candidate_status { last_received { term: 1 index: 399 } } ignore_live_leader: true dest_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e"
I20240501 08:12:29.288093 1291 raft_consensus.cc:3037] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:29.292361 1291 raft_consensus.cc:2454] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e6c8e1eb974b5c9c8b7e109b214576 in term 2.
I20240501 08:12:29.300838 1291 raft_consensus.cc:1261] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 1 index: 399. Preceding OpId from leader: term: 2 index: 401. (index mismatch)
I20240501 08:12:29.350445 1906 consensus_queue.cc:1035] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
W20240501 08:12:30.012826 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:39729, user_credentials={real_user=slave}} blocked reactor thread for 50773.4us
I20240501 08:12:30.284940 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.299746 32569 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:30.344802 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.387351 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:30.402801 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:30.484136 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:30.727916 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:32.003338 32454 consensus_queue.cc:237] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 435, Committed index: 435, Last appended: 2.435, Last appended by leader: 399, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 436 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:32.009948 1293 raft_consensus.cc:1261] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 2 index: 435. Preceding OpId from leader: term: 2 index: 436. (index mismatch)
I20240501 08:12:32.011044 2812 consensus_queue.cc:1035] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 436, Last known committed idx: 435, Time since last communication: 0.000s
I20240501 08:12:32.020071 2812 raft_consensus.cc:2932] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Committing config change with OpId 2.436: config changed from index 95 to 436, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 436 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:32.021306 1293 raft_consensus.cc:2932] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Committing config change with OpId 2.436: config changed from index 95 to 436, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 436 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:32.028918 32181 catalog_manager.cc:5484] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 reported cstate change: config changed from index 95 to 436, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New cstate: current_term: 2 leader_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" committed_config { opid_index: 436 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:32.168144 32569 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a25d38a68a424ebf78f02d2e0750b1 with delete_type TABLET_DATA_TOMBSTONED (TS 12e44d7017614f88b3205111d85de021 not found in new config with opid_index 436) from {username='slave'} at 127.0.0.1:39982
I20240501 08:12:32.195915 2920 tablet_replica.cc:330] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: stopping tablet replica
I20240501 08:12:32.201905 2920 raft_consensus.cc:2229] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:32.205089 2920 raft_consensus.cc:2258] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.230362 2920 ts_tablet_manager.cc:1875] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:32.269307 2920 ts_tablet_manager.cc:1888] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.435
I20240501 08:12:32.269505 2920 log.cc:1196] T 95a25d38a68a424ebf78f02d2e0750b1 P 12e44d7017614f88b3205111d85de021: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/95a25d38a68a424ebf78f02d2e0750b1
I20240501 08:12:32.271100 32173 catalog_manager.cc:4830] TS 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795): tablet 95a25d38a68a424ebf78f02d2e0750b1 (table rebalance_test_table_3 [id=2cc8097a76e54347ad4bc67576658bc9]) successfully deleted
I20240501 08:12:33.316447 32569 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:33.368887 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.403788 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:33.479131 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:33.500488 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.542227 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:33.553697 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:34.223433 32326 consensus_queue.cc:237] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 401, Committed index: 401, Last appended: 1.401, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:34.229195 1292 raft_consensus.cc:1261] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 401. Preceding OpId from leader: term: 1 index: 402. (index mismatch)
I20240501 08:12:34.229959 2895 consensus_queue.cc:1035] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 401, Time since last communication: 0.000s
I20240501 08:12:34.234957 2929 raft_consensus.cc:2932] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.402: config changed from index 66 to 402, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:34.236402 1292 raft_consensus.cc:2932] T 2e794c151e6e40dbb612e64853415d9a P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Committing config change with OpId 1.402: config changed from index 66 to 402, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:34.245446 32182 catalog_manager.cc:5484] T 2e794c151e6e40dbb612e64853415d9a P 2219c1853e76474799f93fed12cdc12d reported cstate change: config changed from index 66 to 402, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:34.349913 32439 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e794c151e6e40dbb612e64853415d9a with delete_type TABLET_DATA_TOMBSTONED (TS b3e6c8e1eb974b5c9c8b7e109b214576 not found in new config with opid_index 402) from {username='slave'} at 127.0.0.1:55378
I20240501 08:12:34.372567 3017 tablet_replica.cc:330] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576: stopping tablet replica
I20240501 08:12:34.372838 3017 raft_consensus.cc:2229] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:34.373122 3017 raft_consensus.cc:2258] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:34.374788 3017 ts_tablet_manager.cc:1875] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:34.394738 3017 ts_tablet_manager.cc:1888] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.401
I20240501 08:12:34.400570 3017 log.cc:1196] T 2e794c151e6e40dbb612e64853415d9a P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/2e794c151e6e40dbb612e64853415d9a
I20240501 08:12:34.403409 32172 catalog_manager.cc:4830] TS b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875): tablet 2e794c151e6e40dbb612e64853415d9a (table rebalance_test_table_1 [id=3910e4f153974479aa62dc08cd2c1ac3]) successfully deleted
W20240501 08:12:34.696291 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:39729, user_credentials={real_user=slave}} blocked reactor thread for 178746us
I20240501 08:12:35.889500 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:35.929623 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:35.979100 32569 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:35.981263 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:35.999680 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:36.024292 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:36.034526 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:36.932565 32587 consensus_queue.cc:237] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 430, Committed index: 430, Last appended: 1.430, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:36.934376 1422 raft_consensus.cc:1261] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 430. Preceding OpId from leader: term: 1 index: 431. (index mismatch)
I20240501 08:12:36.935238 2384 consensus_queue.cc:1035] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 431, Last known committed idx: 430, Time since last communication: 0.000s
I20240501 08:12:36.939985 3038 raft_consensus.cc:2932] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.431: config changed from index 57 to 431, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:36.955209 32181 catalog_manager.cc:5484] T adbf537bdcdf4443a94799340aab4205 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index 57 to 431, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:36.955451 1421 raft_consensus.cc:2932] T adbf537bdcdf4443a94799340aab4205 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Committing config change with OpId 1.431: config changed from index 57 to 431, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:37.034721 32306 tablet_service.cc:1514] Processing DeleteTablet for tablet adbf537bdcdf4443a94799340aab4205 with delete_type TABLET_DATA_TOMBSTONED (TS 2219c1853e76474799f93fed12cdc12d not found in new config with opid_index 431) from {username='slave'} at 127.0.0.1:60500
I20240501 08:12:37.057031 3117 tablet_replica.cc:330] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d: stopping tablet replica
I20240501 08:12:37.057435 3117 raft_consensus.cc:2229] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:37.057770 3117 raft_consensus.cc:2258] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:37.061657 3117 ts_tablet_manager.cc:1875] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:37.070214 3117 ts_tablet_manager.cc:1888] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.430
I20240501 08:12:37.070451 3117 log.cc:1196] T adbf537bdcdf4443a94799340aab4205 P 2219c1853e76474799f93fed12cdc12d: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/adbf537bdcdf4443a94799340aab4205
I20240501 08:12:37.071465 32171 catalog_manager.cc:4830] TS 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223): tablet adbf537bdcdf4443a94799340aab4205 (table rebalance_test_table_4 [id=e2d2d2401bec456884eaceee2e5edf67]) successfully deleted
I20240501 08:12:37.737495 32569 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:37.797238 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:38.016623 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:38.024498 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:38.035465 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:38.086591 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:38.123787 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:12:38.173252 32263 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:39729, user_credentials={real_user=slave}} blocked reactor thread for 44382.6us
W20240501 08:12:38.323062 1489 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:39729, user_credentials={real_user=slave}} blocked reactor thread for 56076.8us
I20240501 08:12:38.692548 32328 consensus_queue.cc:237] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 495, Committed index: 495, Last appended: 1.495, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:38.695322 1159 raft_consensus.cc:1261] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 495. Preceding OpId from leader: term: 1 index: 496. (index mismatch)
I20240501 08:12:38.697259 2985 consensus_queue.cc:1035] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 496, Last known committed idx: 495, Time since last communication: 0.000s
I20240501 08:12:38.704476 2985 raft_consensus.cc:2932] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.496: config changed from index 70 to 496, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:38.710011 1159 raft_consensus.cc:2932] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Committing config change with OpId 1.496: config changed from index 70 to 496, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:38.731503 32182 catalog_manager.cc:5484] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d reported cstate change: config changed from index 70 to 496, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 496 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:38.820019 32569 tablet_service.cc:1514] Processing DeleteTablet for tablet b4c8611ac7394bbd96cabee647f5c724 with delete_type TABLET_DATA_TOMBSTONED (TS 12e44d7017614f88b3205111d85de021 not found in new config with opid_index 496) from {username='slave'} at 127.0.0.1:39982
I20240501 08:12:38.858583 3209 tablet_replica.cc:330] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021: stopping tablet replica
I20240501 08:12:38.859323 3209 raft_consensus.cc:2229] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:38.859611 3209 raft_consensus.cc:2258] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:38.861994 3209 ts_tablet_manager.cc:1875] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.886158 3209 ts_tablet_manager.cc:1888] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.495
I20240501 08:12:38.887800 3209 log.cc:1196] T b4c8611ac7394bbd96cabee647f5c724 P 12e44d7017614f88b3205111d85de021: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/b4c8611ac7394bbd96cabee647f5c724
I20240501 08:12:38.892135 32173 catalog_manager.cc:4830] TS 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795): tablet b4c8611ac7394bbd96cabee647f5c724 (table rebalance_test_table_4 [id=e2d2d2401bec456884eaceee2e5edf67]) successfully deleted
I20240501 08:12:39.535264 32569 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:39.646900 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:39.683277 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:39.694530 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:39.695928 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:39.700453 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:39.723739 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:40.481103 32587 consensus_queue.cc:237] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 528, Committed index: 528, Last appended: 1.528, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } }
I20240501 08:12:40.484284 1422 raft_consensus.cc:1261] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 528. Preceding OpId from leader: term: 1 index: 529. (index mismatch)
I20240501 08:12:40.484889 3172 consensus_queue.cc:1035] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 529, Last known committed idx: 528, Time since last communication: 0.000s
I20240501 08:12:40.495842 3038 raft_consensus.cc:2932] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.529: config changed from index 98 to 529, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:40.502511 32183 catalog_manager.cc:5484] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index 98 to 529, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:40.502660 1421 raft_consensus.cc:2932] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Committing config change with OpId 1.529: config changed from index 98 to 529, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } }
I20240501 08:12:40.609915 32306 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbd5afc230c41509774812f082f97f0 with delete_type TABLET_DATA_TOMBSTONED (TS 2219c1853e76474799f93fed12cdc12d not found in new config with opid_index 529) from {username='slave'} at 127.0.0.1:60500
I20240501 08:12:40.633332 3300 tablet_replica.cc:330] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d: stopping tablet replica
I20240501 08:12:40.633705 3300 raft_consensus.cc:2229] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.636500 3300 raft_consensus.cc:2258] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.644598 3300 ts_tablet_manager.cc:1875] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.664702 3300 ts_tablet_manager.cc:1888] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.528
I20240501 08:12:40.664899 3300 log.cc:1196] T 5dbd5afc230c41509774812f082f97f0 P 2219c1853e76474799f93fed12cdc12d: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/5dbd5afc230c41509774812f082f97f0
I20240501 08:12:40.666157 32171 catalog_manager.cc:4830] TS 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223): tablet 5dbd5afc230c41509774812f082f97f0 (table rebalance_test_table_2 [id=821c266bb4a949cf878e5f7a6df80abf]) successfully deleted
I20240501 08:12:41.450901 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:41.515959 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:41.516041 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:41.597522 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:41.627230 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:41.628556 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:41.681545 32569 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:41.928135 32587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ad5a9e14873742caa50ec838cca1c715"
dest_uuid: "12e44d7017614f88b3205111d85de021"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49122
I20240501 08:12:41.928390 32587 raft_consensus.cc:595] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:41.937517 3106 raft_consensus.cc:979] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: : Instructing follower b3e6c8e1eb974b5c9c8b7e109b214576 to start an election
I20240501 08:12:41.937709 3106 raft_consensus.cc:1067] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Signalling peer b3e6c8e1eb974b5c9c8b7e109b214576 to start an election
I20240501 08:12:41.945319 32459 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ad5a9e14873742caa50ec838cca1c715"
dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
from {username='slave'} at 127.25.107.3:41173
I20240501 08:12:41.945741 32459 raft_consensus.cc:482] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:41.945914 32459 raft_consensus.cc:3037] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:41.952842 32459 raft_consensus.cc:504] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:41.953635 32459 leader_election.cc:288] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 election: Requested vote from peers 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795), c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501)
I20240501 08:12:41.954246 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5a9e14873742caa50ec838cca1c715" candidate_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" candidate_term: 2 candidate_status { last_received { term: 1 index: 549 } } ignore_live_leader: true dest_uuid: "12e44d7017614f88b3205111d85de021"
I20240501 08:12:41.954511 32587 raft_consensus.cc:3032] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:41.954586 32587 raft_consensus.cc:729] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Running, Role: LEADER
I20240501 08:12:41.954843 32587 consensus_queue.cc:260] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 549, Committed index: 549, Last appended: 1.549, Last appended by leader: 549, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:41.955211 32587 raft_consensus.cc:3037] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:41.958171 32587 raft_consensus.cc:2454] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e6c8e1eb974b5c9c8b7e109b214576 in term 2.
I20240501 08:12:41.959059 32394 leader_election.cc:302] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:12:41.959429 3284 raft_consensus.cc:2781] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:41.959582 3284 raft_consensus.cc:686] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Becoming Leader. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Running, Role: LEADER
I20240501 08:12:41.959888 3284 consensus_queue.cc:237] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 549, Committed index: 549, Last appended: 1.549, Last appended by leader: 549, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:41.963456 32182 catalog_manager.cc:5484] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 reported cstate change: term changed from 1 to 2, leader changed from 12e44d7017614f88b3205111d85de021 (127.25.107.3) to b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2). New cstate: current_term: 2 leader_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
W20240501 08:12:42.021184 32549 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57638: Illegal state: replica 12e44d7017614f88b3205111d85de021 is not leader of this config: current role FOLLOWER
I20240501 08:12:42.029146 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5a9e14873742caa50ec838cca1c715" candidate_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" candidate_term: 2 candidate_status { last_received { term: 1 index: 549 } } ignore_live_leader: true dest_uuid: "c5e88f3df73d4d4f996304b6df7e6448"
I20240501 08:12:42.029636 1555 raft_consensus.cc:3037] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:42.050174 1555 raft_consensus.cc:2454] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e6c8e1eb974b5c9c8b7e109b214576 in term 2.
I20240501 08:12:42.061445 32587 raft_consensus.cc:1261] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 1 index: 549. Preceding OpId from leader: term: 2 index: 551. (index mismatch)
I20240501 08:12:42.062075 3284 consensus_queue.cc:1035] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 550, Last known committed idx: 549, Time since last communication: 0.000s
I20240501 08:12:42.073766 1556 raft_consensus.cc:1261] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 1 index: 549. Preceding OpId from leader: term: 2 index: 551. (index mismatch)
I20240501 08:12:42.077111 3341 consensus_queue.cc:1035] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 550, Last known committed idx: 549, Time since last communication: 0.000s
I20240501 08:12:42.941884 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:42.964479 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:42.979097 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:43.012611 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:43.016000 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:43.112360 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:43.170576 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:43.661028 32459 consensus_queue.cc:237] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 574, Committed index: 574, Last appended: 2.575, Last appended by leader: 549, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 576 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:43.664407 1555 raft_consensus.cc:1261] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 2 index: 575. Preceding OpId from leader: term: 2 index: 576. (index mismatch)
I20240501 08:12:43.666627 3377 consensus_queue.cc:1035] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 576, Last known committed idx: 575, Time since last communication: 0.000s
I20240501 08:12:43.670964 3284 raft_consensus.cc:2932] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Committing config change with OpId 2.576: config changed from index 79 to 576, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 576 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:43.680384 32183 catalog_manager.cc:5484] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 reported cstate change: config changed from index 79 to 576, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New cstate: current_term: 2 leader_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" committed_config { opid_index: 576 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:43.686866 1555 raft_consensus.cc:2932] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Committing config change with OpId 2.576: config changed from index 79 to 576, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 576 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:43.701603 32569 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5a9e14873742caa50ec838cca1c715 with delete_type TABLET_DATA_TOMBSTONED (TS 12e44d7017614f88b3205111d85de021 not found in new config with opid_index 576) from {username='slave'} at 127.0.0.1:39982
I20240501 08:12:43.709039 3460 tablet_replica.cc:330] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: stopping tablet replica
I20240501 08:12:43.709260 3460 raft_consensus.cc:2229] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.709465 3460 raft_consensus.cc:2258] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.728961 3460 ts_tablet_manager.cc:1875] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.753592 3460 ts_tablet_manager.cc:1888] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.575
I20240501 08:12:43.753863 3460 log.cc:1196] T ad5a9e14873742caa50ec838cca1c715 P 12e44d7017614f88b3205111d85de021: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/ad5a9e14873742caa50ec838cca1c715
I20240501 08:12:43.755645 32173 catalog_manager.cc:4830] TS 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795): tablet ad5a9e14873742caa50ec838cca1c715 (table rebalance_test_table_1 [id=3910e4f153974479aa62dc08cd2c1ac3]) successfully deleted
I20240501 08:12:44.583240 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:44.586838 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:44.587461 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:44.591158 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:44.611167 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:44.741658 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:44.744592 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:45.349803 32588 consensus_queue.cc:237] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 655, Committed index: 655, Last appended: 1.655, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 656 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:45.358162 1158 raft_consensus.cc:1261] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 655. Preceding OpId from leader: term: 1 index: 656. (index mismatch)
I20240501 08:12:45.359246 2384 consensus_queue.cc:1035] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 656, Last known committed idx: 654, Time since last communication: 0.000s
I20240501 08:12:45.363015 3426 raft_consensus.cc:2932] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.656: config changed from index 128 to 656, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 656 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:45.364979 1159 raft_consensus.cc:2932] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Committing config change with OpId 1.656: config changed from index 128 to 656, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 656 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:45.370143 32181 catalog_manager.cc:5484] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index 128 to 656, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 656 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:45.450086 32439 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c58a563cf94527895a675a1e729fe8 with delete_type TABLET_DATA_TOMBSTONED (TS b3e6c8e1eb974b5c9c8b7e109b214576 not found in new config with opid_index 656) from {username='slave'} at 127.0.0.1:55378
I20240501 08:12:45.465680 3545 tablet_replica.cc:330] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576: stopping tablet replica
I20240501 08:12:45.466760 3545 raft_consensus.cc:2229] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.467119 3545 raft_consensus.cc:2258] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.470407 3545 ts_tablet_manager.cc:1875] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:45.500149 3545 ts_tablet_manager.cc:1888] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.655
I20240501 08:12:45.500392 3545 log.cc:1196] T 50c58a563cf94527895a675a1e729fe8 P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/50c58a563cf94527895a675a1e729fe8
I20240501 08:12:45.502012 32172 catalog_manager.cc:4830] TS b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875): tablet 50c58a563cf94527895a675a1e729fe8 (table rebalance_test_table_3 [id=2cc8097a76e54347ad4bc67576658bc9]) successfully deleted
W20240501 08:12:46.656322 32393 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:42813, user_credentials={real_user=slave}} blocked reactor thread for 86479.2us
I20240501 08:12:47.450016 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:47.459733 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:47.477857 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:47.496773 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:47.497215 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:47.517766 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:47.580430 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:47.986721 32325 consensus_queue.cc:237] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 616, Committed index: 616, Last appended: 1.616, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:48.004057 32456 raft_consensus.cc:1261] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 616. Preceding OpId from leader: term: 1 index: 617. (index mismatch)
I20240501 08:12:48.005836 3467 consensus_queue.cc:1035] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 617, Last known committed idx: 616, Time since last communication: 0.000s
I20240501 08:12:48.009294 3691 raft_consensus.cc:2932] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.617: config changed from index -1 to 617, VOTER c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7) added. New config: { opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:48.010133 32456 raft_consensus.cc:2932] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Committing config change with OpId 1.617: config changed from index -1 to 617, VOTER c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7) added. New config: { opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:48.015081 32182 catalog_manager.cc:5484] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d reported cstate change: config changed from index -1 to 617, VOTER c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
W20240501 08:12:48.033612 32261 consensus_peers.cc:485] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d -> Peer c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501): Couldn't send request to peer c5e88f3df73d4d4f996304b6df7e6448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9d4354546c040d6b971ed6fc3b02c76. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.454609 3706 ts_tablet_manager.cc:902] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448: Initiating tablet copy from peer 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:48.692986 32456 consensus_queue.cc:237] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 739, Committed index: 739, Last appended: 2.740, Last appended by leader: 399, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 741 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:48.695715 1291 raft_consensus.cc:1261] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 2 index: 740. Preceding OpId from leader: term: 2 index: 741. (index mismatch)
W20240501 08:12:48.695905 32392 consensus_peers.cc:485] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 -> Peer c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501): Couldn't send request to peer c5e88f3df73d4d4f996304b6df7e6448. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95a25d38a68a424ebf78f02d2e0750b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.696435 3284 consensus_queue.cc:1035] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 741, Last known committed idx: 739, Time since last communication: 0.000s
I20240501 08:12:48.699954 3556 raft_consensus.cc:2932] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Committing config change with OpId 2.741: config changed from index 436 to 741, VOTER c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7) added. New config: { opid_index: 741 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:48.702982 1291 raft_consensus.cc:2932] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Committing config change with OpId 2.741: config changed from index 436 to 741, VOTER c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7) added. New config: { opid_index: 741 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:48.706784 32183 catalog_manager.cc:5484] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 reported cstate change: config changed from index 436 to 741, VOTER c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7) added. New cstate: current_term: 2 leader_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" committed_config { opid_index: 741 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:48.741251 3706 tablet_copy_client.cc:308] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:33223
W20240501 08:12:48.782590 32523 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:44049, user_credentials={real_user=slave}} blocked reactor thread for 62394.3us
I20240501 08:12:48.850852 32348 tablet_copy_service.cc:140] P 2219c1853e76474799f93fed12cdc12d: Received BeginTabletCopySession request for tablet a9d4354546c040d6b971ed6fc3b02c76 from peer c5e88f3df73d4d4f996304b6df7e6448 ({username='slave'} at 127.25.107.7:45653)
I20240501 08:12:48.851038 32348 tablet_copy_service.cc:161] P 2219c1853e76474799f93fed12cdc12d: Beginning new tablet copy session on tablet a9d4354546c040d6b971ed6fc3b02c76 from peer c5e88f3df73d4d4f996304b6df7e6448 at {username='slave'} at 127.25.107.7:45653: session id = c5e88f3df73d4d4f996304b6df7e6448-a9d4354546c040d6b971ed6fc3b02c76
I20240501 08:12:48.854085 32348 tablet_copy_source_session.cc:205] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.856282 3706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d4354546c040d6b971ed6fc3b02c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.872481 3706 tablet_copy_client.cc:783] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.872913 3706 tablet_copy_client.cc:647] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.910566 3706 tablet_copy_client.cc:523] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.917419 3706 tablet_bootstrap.cc:492] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap starting.
I20240501 08:12:49.241341 3755 ts_tablet_manager.cc:902] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448: Initiating tablet copy from peer b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:12:49.292147 3755 tablet_copy_client.cc:308] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46875
I20240501 08:12:49.293766 32479 tablet_copy_service.cc:140] P b3e6c8e1eb974b5c9c8b7e109b214576: Received BeginTabletCopySession request for tablet 95a25d38a68a424ebf78f02d2e0750b1 from peer c5e88f3df73d4d4f996304b6df7e6448 ({username='slave'} at 127.25.107.7:56997)
I20240501 08:12:49.294011 32479 tablet_copy_service.cc:161] P b3e6c8e1eb974b5c9c8b7e109b214576: Beginning new tablet copy session on tablet 95a25d38a68a424ebf78f02d2e0750b1 from peer c5e88f3df73d4d4f996304b6df7e6448 at {username='slave'} at 127.25.107.7:56997: session id = c5e88f3df73d4d4f996304b6df7e6448-95a25d38a68a424ebf78f02d2e0750b1
I20240501 08:12:49.316962 32479 tablet_copy_source_session.cc:205] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:49.379588 3755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a25d38a68a424ebf78f02d2e0750b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:49.403681 3755 tablet_copy_client.cc:783] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:49.404078 3755 tablet_copy_client.cc:647] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:49.422863 3755 tablet_copy_client.cc:523] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:49.435464 3755 tablet_bootstrap.cc:492] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap starting.
I20240501 08:12:49.453543 32325 consensus_queue.cc:237] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 711, Committed index: 711, Last appended: 1.711, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 712 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
W20240501 08:12:49.456686 32262 consensus_peers.cc:485] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d -> Peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5:42813): Couldn't send request to peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4ec90b4ddce408a81f588fa7546e0dd. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:49.457013 1161 raft_consensus.cc:1261] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 711. Preceding OpId from leader: term: 1 index: 712. (index mismatch)
I20240501 08:12:49.457659 3546 consensus_queue.cc:1035] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 712, Last known committed idx: 711, Time since last communication: 0.000s
I20240501 08:12:49.460696 3546 raft_consensus.cc:2932] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.712: config changed from index 293 to 712, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New config: { opid_index: 712 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:49.461272 1161 raft_consensus.cc:2932] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Committing config change with OpId 1.712: config changed from index 293 to 712, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New config: { opid_index: 712 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:12:49.469056 32182 catalog_manager.cc:5484] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d reported cstate change: config changed from index 293 to 712, VOTER 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 712 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
W20240501 08:12:49.595233 32393 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:42813, user_credentials={real_user=slave}} blocked reactor thread for 49720.6us
I20240501 08:12:50.024340 3787 ts_tablet_manager.cc:902] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Initiating tablet copy from peer 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
W20240501 08:12:50.252236 32523 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:44049, user_credentials={real_user=slave}} blocked reactor thread for 47999.2us
I20240501 08:12:50.273563 32325 consensus_queue.cc:237] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 602, Committed index: 602, Last appended: 1.602, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
W20240501 08:12:50.277714 32263 consensus_peers.cc:485] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d -> Peer 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729): Couldn't send request to peer 489550c4b21146918c1c9618be65b4ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 789da5b22732463da07c34e8716bbb7e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:50.279747 32459 raft_consensus.cc:1261] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 602. Preceding OpId from leader: term: 1 index: 603. (index mismatch)
I20240501 08:12:50.282052 3546 consensus_queue.cc:1035] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 603, Last known committed idx: 602, Time since last communication: 0.000s
I20240501 08:12:50.284559 3546 raft_consensus.cc:2932] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.603: config changed from index -1 to 603, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New config: { opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:50.285563 32456 raft_consensus.cc:2932] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Committing config change with OpId 1.603: config changed from index -1 to 603, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New config: { opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:50.294562 32182 catalog_manager.cc:5484] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d reported cstate change: config changed from index -1 to 603, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) added. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:12:50.519960 3787 tablet_copy_client.cc:308] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:33223
I20240501 08:12:50.520874 32348 tablet_copy_service.cc:140] P 2219c1853e76474799f93fed12cdc12d: Received BeginTabletCopySession request for tablet f4ec90b4ddce408a81f588fa7546e0dd from peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e ({username='slave'} at 127.25.107.5:35565)
I20240501 08:12:50.521044 32348 tablet_copy_service.cc:161] P 2219c1853e76474799f93fed12cdc12d: Beginning new tablet copy session on tablet f4ec90b4ddce408a81f588fa7546e0dd from peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e at {username='slave'} at 127.25.107.5:35565: session id = 92f76e9a8b5c4bc9b49900f5ca6b4f6e-f4ec90b4ddce408a81f588fa7546e0dd
I20240501 08:12:50.523730 32348 tablet_copy_source_session.cc:205] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:50.536506 3706 tablet_bootstrap.cc:492] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap replayed 1/1 log segments. Stats: ops{read=626 overwritten=0 applied=626 ignored=0} inserts{seen=1298 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:50.537091 3706 tablet_bootstrap.cc:492] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap complete.
I20240501 08:12:50.537526 3706 ts_tablet_manager.cc:1367] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448: Time spent bootstrapping tablet: real 1.620s user 0.699s sys 0.122s
I20240501 08:12:50.538465 3706 raft_consensus.cc:348] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:50.538826 3706 raft_consensus.cc:729] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e88f3df73d4d4f996304b6df7e6448, State: Initialized, Role: FOLLOWER
I20240501 08:12:50.539173 3706 consensus_queue.cc:260] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 626, Last appended: 1.626, Last appended by leader: 626, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:50.539846 3706 ts_tablet_manager.cc:1398] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:50.540491 32348 tablet_copy_service.cc:342] P 2219c1853e76474799f93fed12cdc12d: Request end of tablet copy session c5e88f3df73d4d4f996304b6df7e6448-a9d4354546c040d6b971ed6fc3b02c76 received from {username='slave'} at 127.25.107.7:45653
I20240501 08:12:50.540624 32348 tablet_copy_service.cc:433] P 2219c1853e76474799f93fed12cdc12d: ending tablet copy session c5e88f3df73d4d4f996304b6df7e6448-a9d4354546c040d6b971ed6fc3b02c76 on tablet a9d4354546c040d6b971ed6fc3b02c76 with peer c5e88f3df73d4d4f996304b6df7e6448
I20240501 08:12:50.585942 3787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ec90b4ddce408a81f588fa7546e0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:50.607184 3787 tablet_copy_client.cc:783] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:50.607610 3787 tablet_copy_client.cc:647] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:50.613677 3787 tablet_copy_client.cc:523] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:50.627945 3787 tablet_bootstrap.cc:492] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap starting.
I20240501 08:12:50.720768 1555 raft_consensus.cc:1203] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.616->[1.617-1.656] Dedup: 1.626->[1.627-1.656]
I20240501 08:12:50.772461 3820 ts_tablet_manager.cc:902] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff: Initiating tablet copy from peer 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:12:50.995165 3755 tablet_bootstrap.cc:492] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap replayed 1/1 log segments. Stats: ops{read=756 overwritten=0 applied=755 ignored=0} inserts{seen=1437 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:12:50.995812 3755 tablet_bootstrap.cc:492] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448: Bootstrap complete.
I20240501 08:12:50.996239 3755 ts_tablet_manager.cc:1367] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448: Time spent bootstrapping tablet: real 1.562s user 0.853s sys 0.062s
I20240501 08:12:50.997153 3755 raft_consensus.cc:348] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 741 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:50.997929 3755 raft_consensus.cc:729] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e88f3df73d4d4f996304b6df7e6448, State: Initialized, Role: FOLLOWER
I20240501 08:12:50.998292 3755 consensus_queue.cc:260] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 755, Last appended: 2.756, Last appended by leader: 756, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 741 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:50.999193 3755 ts_tablet_manager.cc:1398] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:51.000041 32479 tablet_copy_service.cc:342] P b3e6c8e1eb974b5c9c8b7e109b214576: Request end of tablet copy session c5e88f3df73d4d4f996304b6df7e6448-95a25d38a68a424ebf78f02d2e0750b1 received from {username='slave'} at 127.25.107.7:56997
I20240501 08:12:51.000217 32479 tablet_copy_service.cc:433] P b3e6c8e1eb974b5c9c8b7e109b214576: ending tablet copy session c5e88f3df73d4d4f996304b6df7e6448-95a25d38a68a424ebf78f02d2e0750b1 on tablet 95a25d38a68a424ebf78f02d2e0750b1 with peer c5e88f3df73d4d4f996304b6df7e6448
I20240501 08:12:51.076854 3820 tablet_copy_client.cc:308] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:33223
I20240501 08:12:51.078891 32348 tablet_copy_service.cc:140] P 2219c1853e76474799f93fed12cdc12d: Received BeginTabletCopySession request for tablet 789da5b22732463da07c34e8716bbb7e from peer 489550c4b21146918c1c9618be65b4ff ({username='slave'} at 127.25.107.4:43313)
I20240501 08:12:51.079099 32348 tablet_copy_service.cc:161] P 2219c1853e76474799f93fed12cdc12d: Beginning new tablet copy session on tablet 789da5b22732463da07c34e8716bbb7e from peer 489550c4b21146918c1c9618be65b4ff at {username='slave'} at 127.25.107.4:43313: session id = 489550c4b21146918c1c9618be65b4ff-789da5b22732463da07c34e8716bbb7e
I20240501 08:12:51.082108 32348 tablet_copy_source_session.cc:205] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:51.084599 3820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789da5b22732463da07c34e8716bbb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:51.183809 3820 tablet_copy_client.cc:783] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:51.186748 1556 raft_consensus.cc:1203] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.740->[2.741-2.788] Dedup: 2.756->[2.757-2.788]
I20240501 08:12:51.188673 3820 tablet_copy_client.cc:647] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:51.193804 3820 tablet_copy_client.cc:523] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:51.200278 3820 tablet_bootstrap.cc:492] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff: Bootstrap starting.
W20240501 08:12:51.845382 32523 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:44049, user_credentials={real_user=slave}} blocked reactor thread for 88490.3us
W20240501 08:12:51.901206 32523 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:44049, user_credentials={real_user=slave}} blocked reactor thread for 49766.8us
W20240501 08:12:52.059507 32523 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:42813, user_credentials={real_user=slave}} blocked reactor thread for 50865.1us
I20240501 08:12:52.085539 3787 tablet_bootstrap.cc:492] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=718 overwritten=0 applied=718 ignored=0} inserts{seen=1347 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:52.086103 3787 tablet_bootstrap.cc:492] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Bootstrap complete.
I20240501 08:12:52.086946 3787 ts_tablet_manager.cc:1367] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Time spent bootstrapping tablet: real 1.459s user 0.820s sys 0.067s
I20240501 08:12:52.088562 3787 raft_consensus.cc:348] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 712 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:52.088912 3787 raft_consensus.cc:729] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f76e9a8b5c4bc9b49900f5ca6b4f6e, State: Initialized, Role: FOLLOWER
I20240501 08:12:52.089231 3787 consensus_queue.cc:260] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 718, Last appended: 1.718, Last appended by leader: 718, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 712 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:52.099481 3787 ts_tablet_manager.cc:1398] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Time spent starting tablet: real 0.012s user 0.002s sys 0.008s
I20240501 08:12:52.101573 32348 tablet_copy_service.cc:342] P 2219c1853e76474799f93fed12cdc12d: Request end of tablet copy session 92f76e9a8b5c4bc9b49900f5ca6b4f6e-f4ec90b4ddce408a81f588fa7546e0dd received from {username='slave'} at 127.25.107.5:35565
I20240501 08:12:52.101722 32348 tablet_copy_service.cc:433] P 2219c1853e76474799f93fed12cdc12d: ending tablet copy session 92f76e9a8b5c4bc9b49900f5ca6b4f6e-f4ec90b4ddce408a81f588fa7546e0dd on tablet f4ec90b4ddce408a81f588fa7546e0dd with peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e
I20240501 08:12:52.260318 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:52.327769 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:52.410133 1293 raft_consensus.cc:1203] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.711->[1.712-1.728] Dedup: 1.718->[1.719-1.728]
I20240501 08:12:52.462803 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:52.563463 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:52.581837 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:52.687604 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:52.725687 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:12:53.058835 32262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:44049, user_credentials={real_user=slave}} blocked reactor thread for 57581.2us
I20240501 08:12:53.140974 3820 tablet_bootstrap.cc:492] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=613 overwritten=0 applied=613 ignored=0} inserts{seen=1285 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:53.143354 3820 tablet_bootstrap.cc:492] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff: Bootstrap complete.
I20240501 08:12:53.143752 3820 ts_tablet_manager.cc:1367] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff: Time spent bootstrapping tablet: real 1.944s user 0.730s sys 0.083s
I20240501 08:12:53.144568 3820 raft_consensus.cc:348] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:12:53.144881 3820 raft_consensus.cc:729] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489550c4b21146918c1c9618be65b4ff, State: Initialized, Role: FOLLOWER
I20240501 08:12:53.145234 3820 consensus_queue.cc:260] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 613, Last appended: 1.613, Last appended by leader: 613, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
W20240501 08:12:53.210135 32263 consensus_peers.cc:485] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d -> Peer 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729): Couldn't send request to peer 489550c4b21146918c1c9618be65b4ff. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 08:12:53.371435 3820 ts_tablet_manager.cc:1398] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff: Time spent starting tablet: real 0.227s user 0.025s sys 0.082s
I20240501 08:12:53.376209 32348 tablet_copy_service.cc:342] P 2219c1853e76474799f93fed12cdc12d: Request end of tablet copy session 489550c4b21146918c1c9618be65b4ff-789da5b22732463da07c34e8716bbb7e received from {username='slave'} at 127.25.107.4:43313
I20240501 08:12:53.376386 32348 tablet_copy_service.cc:433] P 2219c1853e76474799f93fed12cdc12d: ending tablet copy session 489550c4b21146918c1c9618be65b4ff-789da5b22732463da07c34e8716bbb7e on tablet 789da5b22732463da07c34e8716bbb7e with peer 489550c4b21146918c1c9618be65b4ff
I20240501 08:12:53.577852 1160 raft_consensus.cc:1203] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.602->[1.603-1.634] Dedup: 1.613->[1.614-1.634]
I20240501 08:12:53.941960 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:12:53.942674 32262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:42813, user_credentials={real_user=slave}} blocked reactor thread for 62523.6us
I20240501 08:12:53.981328 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:54.010102 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:54.039145 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:54.079535 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:54.144636 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:54.211750 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:55.533265 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:55.535571 32306 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:55.590687 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:55.599610 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:55.606570 1142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:55.659844 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:55.732573 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:12:55.813483 32263 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:39729, user_credentials={real_user=slave}} blocked reactor thread for 101715us
I20240501 08:12:56.245090 32328 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f4ec90b4ddce408a81f588fa7546e0dd"
dest_uuid: "2219c1853e76474799f93fed12cdc12d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40972
I20240501 08:12:56.245366 32328 raft_consensus.cc:595] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Received request to transfer leadership
W20240501 08:12:56.323199 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.331488 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.339367 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.351559 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.361786 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.372467 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.385810 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.397665 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.411161 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.427237 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.441996 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.466006 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.484440 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.510843 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.552723 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.574131 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.597944 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.619752 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.644502 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.672108 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.700621 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.733888 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
I20240501 08:12:56.759516 3894 raft_consensus.cc:979] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: : Instructing follower 489550c4b21146918c1c9618be65b4ff to start an election
I20240501 08:12:56.759716 3894 raft_consensus.cc:1067] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Signalling peer 489550c4b21146918c1c9618be65b4ff to start an election
W20240501 08:12:56.762442 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
I20240501 08:12:56.762737 1160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f4ec90b4ddce408a81f588fa7546e0dd"
dest_uuid: "489550c4b21146918c1c9618be65b4ff"
from {username='slave'} at 127.25.107.1:49819
I20240501 08:12:56.764549 1160 raft_consensus.cc:482] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:56.764811 1160 raft_consensus.cc:3037] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:12:56.793215 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
W20240501 08:12:56.826783 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
I20240501 08:12:56.828871 1160 raft_consensus.cc:504] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 712 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
W20240501 08:12:56.870636 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Service unavailable: leader transfer in progress
I20240501 08:12:56.891841 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ec90b4ddce408a81f588fa7546e0dd" candidate_uuid: "489550c4b21146918c1c9618be65b4ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 782 } } ignore_live_leader: true dest_uuid: "2219c1853e76474799f93fed12cdc12d"
I20240501 08:12:56.892225 32328 raft_consensus.cc:3032] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:56.892480 32328 raft_consensus.cc:729] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:12:56.892940 32328 consensus_queue.cc:260] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 782, Committed index: 782, Last appended: 1.782, Last appended by leader: 782, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 712 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:56.893429 32328 raft_consensus.cc:3037] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:56.897486 32328 raft_consensus.cc:2454] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 489550c4b21146918c1c9618be65b4ff in term 2.
I20240501 08:12:56.908282 1160 leader_election.cc:288] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [CANDIDATE]: Term 2 election: Requested vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223), 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5:42813)
W20240501 08:12:56.913259 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Illegal state: replica 2219c1853e76474799f93fed12cdc12d is not leader of this config: current role FOLLOWER
I20240501 08:12:56.951058 1095 leader_election.cc:302] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, 489550c4b21146918c1c9618be65b4ff; no voters:
I20240501 08:12:57.005303 4072 raft_consensus.cc:2781] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Leader election won for term 2
W20240501 08:12:57.005169 1095 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.107.1:33223, user_credentials={real_user=slave}} blocked reactor thread for 54222.3us
I20240501 08:12:57.009522 1293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ec90b4ddce408a81f588fa7546e0dd" candidate_uuid: "489550c4b21146918c1c9618be65b4ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 782 } } ignore_live_leader: true dest_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e"
I20240501 08:12:57.009807 1293 raft_consensus.cc:3037] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:57.012682 1293 raft_consensus.cc:2454] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 489550c4b21146918c1c9618be65b4ff in term 2.
I20240501 08:12:57.027170 4072 raft_consensus.cc:686] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 2 LEADER]: Becoming Leader. State: Replica: 489550c4b21146918c1c9618be65b4ff, State: Running, Role: LEADER
I20240501 08:12:57.028039 4072 consensus_queue.cc:237] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 782, Committed index: 782, Last appended: 1.782, Last appended by leader: 782, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 712 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:12:57.051498 32182 catalog_manager.cc:5484] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff reported cstate change: term changed from 1 to 2, leader changed from 2219c1853e76474799f93fed12cdc12d (127.25.107.1) to 489550c4b21146918c1c9618be65b4ff (127.25.107.4). New cstate: current_term: 2 leader_uuid: "489550c4b21146918c1c9618be65b4ff" committed_config { opid_index: 712 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
W20240501 08:12:57.053824 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50546: Illegal state: replica 2219c1853e76474799f93fed12cdc12d is not leader of this config: current role FOLLOWER
I20240501 08:12:57.095618 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:57.148698 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:57.165757 1293 raft_consensus.cc:1261] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Refusing update from remote peer 489550c4b21146918c1c9618be65b4ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 782. Preceding OpId from leader: term: 2 index: 784. (index mismatch)
I20240501 08:12:57.167084 4072 consensus_queue.cc:1035] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 783, Last known committed idx: 782, Time since last communication: 0.000s
I20240501 08:12:57.168498 32325 raft_consensus.cc:1261] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Refusing update from remote peer 489550c4b21146918c1c9618be65b4ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 782. Preceding OpId from leader: term: 2 index: 784. (index mismatch)
I20240501 08:12:57.178397 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:57.182152 4072 consensus_queue.cc:1035] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 783, Last known committed idx: 782, Time since last communication: 0.000s
I20240501 08:12:57.220842 32306 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:57.277626 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:57.327077 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:57.388008 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:58.703451 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:58.820670 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:58.843534 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:58.850203 1273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:58.866613 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:58.894124 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:58.964537 32306 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:59.146762 32459 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "95a25d38a68a424ebf78f02d2e0750b1"
dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60418
I20240501 08:12:59.147015 32459 raft_consensus.cc:595] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Received request to transfer leadership
W20240501 08:12:59.215474 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.221122 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.245445 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.254176 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.268271 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.276538 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.290585 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.321554 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.338305 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.354929 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.370579 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.385466 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.402333 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.421332 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.449396 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.470199 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.488276 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.519037 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.541277 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.565891 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.593272 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.619421 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
W20240501 08:12:59.648285 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
I20240501 08:12:59.655973 4024 raft_consensus.cc:979] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: : Instructing follower 92f76e9a8b5c4bc9b49900f5ca6b4f6e to start an election
I20240501 08:12:59.656520 3828 raft_consensus.cc:1067] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Signalling peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e to start an election
I20240501 08:12:59.662671 1293 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "95a25d38a68a424ebf78f02d2e0750b1"
dest_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e"
from {username='slave'} at 127.25.107.2:42567
I20240501 08:12:59.663311 1293 raft_consensus.cc:482] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:59.666172 1293 raft_consensus.cc:3037] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:59.676604 1293 raft_consensus.cc:504] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 741 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
W20240501 08:12:59.681397 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Service unavailable: leader transfer in progress
I20240501 08:12:59.713591 1293 leader_election.cc:288] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [CANDIDATE]: Term 3 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875), c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501)
I20240501 08:12:59.751091 32456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a25d38a68a424ebf78f02d2e0750b1" candidate_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" candidate_term: 3 candidate_status { last_received { term: 2 index: 928 } } ignore_live_leader: true dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:12:59.751340 32456 raft_consensus.cc:3032] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:12:59.751435 32456 raft_consensus.cc:729] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Running, Role: LEADER
I20240501 08:12:59.751754 32456 consensus_queue.cc:260] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 928, Committed index: 928, Last appended: 2.928, Last appended by leader: 928, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 741 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:59.752185 32456 raft_consensus.cc:3037] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:59.753476 1556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a25d38a68a424ebf78f02d2e0750b1" candidate_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" candidate_term: 3 candidate_status { last_received { term: 2 index: 928 } } ignore_live_leader: true dest_uuid: "c5e88f3df73d4d4f996304b6df7e6448"
I20240501 08:12:59.753705 1556 raft_consensus.cc:3037] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:12:59.757230 1556 raft_consensus.cc:2454] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f76e9a8b5c4bc9b49900f5ca6b4f6e in term 3.
I20240501 08:12:59.760176 1226 leader_election.cc:302] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f76e9a8b5c4bc9b49900f5ca6b4f6e, c5e88f3df73d4d4f996304b6df7e6448; no voters:
I20240501 08:12:59.761044 4213 raft_consensus.cc:2781] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 3 FOLLOWER]: Leader election won for term 3
W20240501 08:12:59.766651 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Illegal state: replica b3e6c8e1eb974b5c9c8b7e109b214576 is not leader of this config: current role FOLLOWER
I20240501 08:12:59.765331 4213 raft_consensus.cc:686] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 3 LEADER]: Becoming Leader. State: Replica: 92f76e9a8b5c4bc9b49900f5ca6b4f6e, State: Running, Role: LEADER
I20240501 08:12:59.771545 4213 consensus_queue.cc:237] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 928, Committed index: 928, Last appended: 2.928, Last appended by leader: 928, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 741 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:12:59.777598 32183 catalog_manager.cc:5484] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e reported cstate change: term changed from 2 to 3, leader changed from b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) to 92f76e9a8b5c4bc9b49900f5ca6b4f6e (127.25.107.5). New cstate: current_term: 3 leader_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" committed_config { opid_index: 741 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:12:59.816653 32456 raft_consensus.cc:2454] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f76e9a8b5c4bc9b49900f5ca6b4f6e in term 3.
W20240501 08:12:59.845117 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38974: Illegal state: replica b3e6c8e1eb974b5c9c8b7e109b214576 is not leader of this config: current role FOLLOWER
I20240501 08:12:59.945767 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:59.980496 1555 raft_consensus.cc:1261] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448 [term 3 FOLLOWER]: Refusing update from remote peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 928. Preceding OpId from leader: term: 3 index: 930. (index mismatch)
I20240501 08:12:59.982616 4214 consensus_queue.cc:1035] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 929, Last known committed idx: 928, Time since last communication: 0.000s
I20240501 08:13:00.002070 32456 raft_consensus.cc:1261] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 3 FOLLOWER]: Refusing update from remote peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 928. Preceding OpId from leader: term: 3 index: 930. (index mismatch)
I20240501 08:13:00.011547 4214 consensus_queue.cc:1035] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 929, Last known committed idx: 928, Time since last communication: 0.000s
I20240501 08:13:00.013592 4134 mvcc.cc:205] Tried to move back new op lower bound from 7022801633186590720 to 7022801632357937152. Current Snapshot: MvccSnapshot[applied={T|T < 7022801633186590720}]
I20240501 08:13:00.020588 4109 mvcc.cc:205] Tried to move back new op lower bound from 7022801633186590720 to 7022801632357937152. Current Snapshot: MvccSnapshot[applied={T|T < 7022801633186590720}]
I20240501 08:13:00.038601 32306 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:13:00.046200 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:00.088763 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:00.092181 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:00.124821 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:00.143289 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:01.540180 32306 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:13:01.574928 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:01.612373 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:01.639524 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:01.668359 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:01.687664 1535 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:01.757278 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:02.135337 32328 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9d4354546c040d6b971ed6fc3b02c76"
dest_uuid: "2219c1853e76474799f93fed12cdc12d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41024
I20240501 08:13:02.135610 32328 raft_consensus.cc:595] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Received request to transfer leadership
W20240501 08:13:02.176471 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.211859 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.218366 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.225036 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.242534 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.253537 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.268026 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.281158 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.295300 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.307266 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.333436 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.350093 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.371582 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.394315 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.411347 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.432564 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.452026 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.473731 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.494529 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.522497 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
W20240501 08:13:02.553179 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Service unavailable: leader transfer in progress
I20240501 08:13:02.557992 4016 raft_consensus.cc:979] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: : Instructing follower c5e88f3df73d4d4f996304b6df7e6448 to start an election
I20240501 08:13:02.558329 4124 raft_consensus.cc:1067] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Signalling peer c5e88f3df73d4d4f996304b6df7e6448 to start an election
I20240501 08:13:02.559805 1554 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9d4354546c040d6b971ed6fc3b02c76"
dest_uuid: "c5e88f3df73d4d4f996304b6df7e6448"
from {username='slave'} at 127.25.107.1:50117
I20240501 08:13:02.560101 1554 raft_consensus.cc:482] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:02.560277 1554 raft_consensus.cc:3037] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:02.567224 1554 raft_consensus.cc:504] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:13:02.571838 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4354546c040d6b971ed6fc3b02c76" candidate_uuid: "c5e88f3df73d4d4f996304b6df7e6448" candidate_term: 2 candidate_status { last_received { term: 1 index: 798 } } ignore_live_leader: true dest_uuid: "2219c1853e76474799f93fed12cdc12d"
I20240501 08:13:02.572182 32328 raft_consensus.cc:3032] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:02.572296 32328 raft_consensus.cc:729] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:13:02.572621 32328 consensus_queue.cc:260] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 798, Committed index: 798, Last appended: 1.798, Last appended by leader: 798, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:13:02.572993 32328 raft_consensus.cc:3037] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:02.572919 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4354546c040d6b971ed6fc3b02c76" candidate_uuid: "c5e88f3df73d4d4f996304b6df7e6448" candidate_term: 2 candidate_status { last_received { term: 1 index: 798 } } ignore_live_leader: true dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:13:02.573186 32459 raft_consensus.cc:3037] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:02.574028 1554 leader_election.cc:288] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [CANDIDATE]: Term 2 election: Requested vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223), b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:13:02.576857 32328 raft_consensus.cc:2454] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e88f3df73d4d4f996304b6df7e6448 in term 2.
I20240501 08:13:02.577539 1487 leader_election.cc:302] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, c5e88f3df73d4d4f996304b6df7e6448; no voters:
I20240501 08:13:02.578215 4323 raft_consensus.cc:2781] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:02.583793 32459 raft_consensus.cc:2454] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e88f3df73d4d4f996304b6df7e6448 in term 2.
W20240501 08:13:02.584609 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50566: Illegal state: replica 2219c1853e76474799f93fed12cdc12d is not leader of this config: current role FOLLOWER
I20240501 08:13:02.642436 4323 raft_consensus.cc:686] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 LEADER]: Becoming Leader. State: Replica: c5e88f3df73d4d4f996304b6df7e6448, State: Running, Role: LEADER
I20240501 08:13:02.642902 4323 consensus_queue.cc:237] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 798, Committed index: 798, Last appended: 1.798, Last appended by leader: 798, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:13:02.655364 32180 catalog_manager.cc:5484] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 reported cstate change: term changed from 1 to 2, leader changed from 2219c1853e76474799f93fed12cdc12d (127.25.107.1) to c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7). New cstate: current_term: 2 leader_uuid: "c5e88f3df73d4d4f996304b6df7e6448" committed_config { opid_index: 617 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
W20240501 08:13:02.685927 32418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60448: Illegal state: replica b3e6c8e1eb974b5c9c8b7e109b214576 is not leader of this config: current role FOLLOWER
I20240501 08:13:02.784538 32328 raft_consensus.cc:1261] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Refusing update from remote peer c5e88f3df73d4d4f996304b6df7e6448: Log matching property violated. Preceding OpId in replica: term: 1 index: 798. Preceding OpId from leader: term: 2 index: 800. (index mismatch)
I20240501 08:13:02.785467 4333 consensus_queue.cc:1035] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 799, Last known committed idx: 798, Time since last communication: 0.000s
I20240501 08:13:02.792692 32456 raft_consensus.cc:1261] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Refusing update from remote peer c5e88f3df73d4d4f996304b6df7e6448: Log matching property violated. Preceding OpId in replica: term: 1 index: 798. Preceding OpId from leader: term: 2 index: 800. (index mismatch)
I20240501 08:13:02.793587 4323 consensus_queue.cc:1035] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 799, Last known committed idx: 798, Time since last communication: 0.000s
I20240501 08:13:02.803618 4218 mvcc.cc:205] Tried to move back new op lower bound from 7022801644670115840 to 7022801644145319936. Current Snapshot: MvccSnapshot[applied={T|T < 7022801644670115840 or (T in {7022801644670115840})}]
I20240501 08:13:03.102977 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:13:03.130218 32392 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.1:33223, user_credentials={real_user=slave}} blocked reactor thread for 80055us
I20240501 08:13:03.158674 32306 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:03.239823 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:03.294447 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:03.350183 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:03.369817 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:03.490039 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:13:03.636245 1096 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:42813, user_credentials={real_user=slave}} blocked reactor thread for 76823.8us
I20240501 08:13:04.071555 1556 consensus_queue.cc:237] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 815, Committed index: 815, Last appended: 2.815, Last appended by leader: 798, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 816 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:13:04.073343 32459 raft_consensus.cc:1261] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Refusing update from remote peer c5e88f3df73d4d4f996304b6df7e6448: Log matching property violated. Preceding OpId in replica: term: 2 index: 815. Preceding OpId from leader: term: 2 index: 816. (index mismatch)
I20240501 08:13:04.074674 4323 consensus_queue.cc:1035] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 816, Last known committed idx: 815, Time since last communication: 0.000s
I20240501 08:13:04.084758 4323 raft_consensus.cc:2932] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 LEADER]: Committing config change with OpId 2.816: config changed from index 617 to 816, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 816 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:13:04.085283 32459 raft_consensus.cc:2932] T a9d4354546c040d6b971ed6fc3b02c76 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Committing config change with OpId 2.816: config changed from index 617 to 816, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 816 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:13:04.090771 32181 catalog_manager.cc:5484] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 reported cstate change: config changed from index 617 to 816, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New cstate: current_term: 2 leader_uuid: "c5e88f3df73d4d4f996304b6df7e6448" committed_config { opid_index: 816 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:13:04.286060 32306 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d4354546c040d6b971ed6fc3b02c76 with delete_type TABLET_DATA_TOMBSTONED (TS 2219c1853e76474799f93fed12cdc12d not found in new config with opid_index 816) from {username='slave'} at 127.0.0.1:60500
I20240501 08:13:04.313058 4418 tablet_replica.cc:330] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: stopping tablet replica
I20240501 08:13:04.314054 4418 raft_consensus.cc:2229] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:04.314587 4418 raft_consensus.cc:2258] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:04.324164 4418 ts_tablet_manager.cc:1875] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:04.336594 4418 ts_tablet_manager.cc:1888] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.815
I20240501 08:13:04.336796 4418 log.cc:1196] T a9d4354546c040d6b971ed6fc3b02c76 P 2219c1853e76474799f93fed12cdc12d: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/a9d4354546c040d6b971ed6fc3b02c76
I20240501 08:13:04.338217 32171 catalog_manager.cc:4830] TS 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223): tablet a9d4354546c040d6b971ed6fc3b02c76 (table rebalance_test_table_4 [id=e2d2d2401bec456884eaceee2e5edf67]) successfully deleted
I20240501 08:13:05.453083 32306 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:05.456043 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:05.514084 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:05.543102 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:05.619446 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:05.686847 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:05.786046 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:06.449968 32328 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "789da5b22732463da07c34e8716bbb7e"
dest_uuid: "2219c1853e76474799f93fed12cdc12d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56338
I20240501 08:13:06.450280 32328 raft_consensus.cc:595] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Received request to transfer leadership
W20240501 08:13:06.521015 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.526986 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.533409 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.541453 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.551435 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.565397 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.579056 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.596879 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.609906 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.623245 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.641788 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.661285 32286 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.690078 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.711553 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.741142 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.769446 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.789618 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.815479 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
I20240501 08:13:06.847764 4124 raft_consensus.cc:979] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: : Instructing follower b3e6c8e1eb974b5c9c8b7e109b214576 to start an election
I20240501 08:13:06.847965 4124 raft_consensus.cc:1067] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Signalling peer b3e6c8e1eb974b5c9c8b7e109b214576 to start an election
I20240501 08:13:06.862582 32456 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "789da5b22732463da07c34e8716bbb7e"
dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
from {username='slave'} at 127.25.107.1:48169
I20240501 08:13:06.862828 32456 raft_consensus.cc:482] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:06.862924 32456 raft_consensus.cc:3037] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:06.865818 32456 raft_consensus.cc:504] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
W20240501 08:13:06.867668 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
W20240501 08:13:06.896997 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Service unavailable: leader transfer in progress
I20240501 08:13:06.914048 32328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789da5b22732463da07c34e8716bbb7e" candidate_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" candidate_term: 2 candidate_status { last_received { term: 1 index: 814 } } ignore_live_leader: true dest_uuid: "2219c1853e76474799f93fed12cdc12d"
I20240501 08:13:06.914407 32328 raft_consensus.cc:3032] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:06.914537 32328 raft_consensus.cc:729] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:13:06.914848 32328 consensus_queue.cc:260] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 814, Committed index: 814, Last appended: 1.814, Last appended by leader: 814, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:13:06.915427 32328 raft_consensus.cc:3037] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:06.921247 32285 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50530: Illegal state: replica 2219c1853e76474799f93fed12cdc12d is not leader of this config: current role FOLLOWER
I20240501 08:13:06.928948 32328 raft_consensus.cc:2454] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e6c8e1eb974b5c9c8b7e109b214576 in term 2.
I20240501 08:13:06.930729 32456 leader_election.cc:288] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 election: Requested vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223), 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729)
I20240501 08:13:06.938462 32392 leader_election.cc:302] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:13:06.938935 4416 raft_consensus.cc:2781] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:06.939085 4416 raft_consensus.cc:686] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Becoming Leader. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Running, Role: LEADER
I20240501 08:13:06.939347 4416 consensus_queue.cc:237] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 814, Committed index: 814, Last appended: 1.814, Last appended by leader: 814, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:13:06.958668 32180 catalog_manager.cc:5484] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 reported cstate change: term changed from 1 to 2, leader changed from 2219c1853e76474799f93fed12cdc12d (127.25.107.1) to b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2). New cstate: current_term: 2 leader_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" committed_config { opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:13:06.989195 1159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789da5b22732463da07c34e8716bbb7e" candidate_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" candidate_term: 2 candidate_status { last_received { term: 1 index: 814 } } ignore_live_leader: true dest_uuid: "489550c4b21146918c1c9618be65b4ff"
I20240501 08:13:06.989657 1159 raft_consensus.cc:3037] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:07.008443 32328 raft_consensus.cc:1261] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 1 index: 814. Preceding OpId from leader: term: 2 index: 816. (index mismatch)
I20240501 08:13:07.009353 4367 consensus_queue.cc:1035] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 815, Last known committed idx: 814, Time since last communication: 0.000s
I20240501 08:13:07.030803 1159 raft_consensus.cc:2454] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e6c8e1eb974b5c9c8b7e109b214576 in term 2.
I20240501 08:13:07.031332 1158 raft_consensus.cc:1261] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 1 index: 814. Preceding OpId from leader: term: 2 index: 816. (index mismatch)
I20240501 08:13:07.038941 4416 consensus_queue.cc:1035] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 815, Last known committed idx: 814, Time since last communication: 0.000s
I20240501 08:13:07.127285 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:07.200805 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:07.344583 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:07.353925 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:07.396492 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:07.440028 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:07.510433 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:08.464123 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:08.571630 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:08.697943 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:08.713109 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:08.765072 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:08.831552 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:08.834988 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:09.326781 1160 consensus_queue.cc:237] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 979, Committed index: 979, Last appended: 2.980, Last appended by leader: 782, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 981 OBSOLETE_local: false peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } }
I20240501 08:13:09.328797 1292 raft_consensus.cc:1261] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Refusing update from remote peer 489550c4b21146918c1c9618be65b4ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 980. Preceding OpId from leader: term: 2 index: 981. (index mismatch)
I20240501 08:13:09.329480 4585 consensus_queue.cc:1035] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 981, Last known committed idx: 979, Time since last communication: 0.000s
I20240501 08:13:09.333609 4585 raft_consensus.cc:2932] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff [term 2 LEADER]: Committing config change with OpId 2.981: config changed from index 712 to 981, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 981 OBSOLETE_local: false peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:13:09.346714 1292 raft_consensus.cc:2932] T f4ec90b4ddce408a81f588fa7546e0dd P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 2 FOLLOWER]: Committing config change with OpId 2.981: config changed from index 712 to 981, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 981 OBSOLETE_local: false peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:13:09.348969 32183 catalog_manager.cc:5484] T f4ec90b4ddce408a81f588fa7546e0dd P 489550c4b21146918c1c9618be65b4ff reported cstate change: config changed from index 712 to 981, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New cstate: current_term: 2 leader_uuid: "489550c4b21146918c1c9618be65b4ff" committed_config { opid_index: 981 OBSOLETE_local: false peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } }
I20240501 08:13:09.532298 32306 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ec90b4ddce408a81f588fa7546e0dd with delete_type TABLET_DATA_TOMBSTONED (TS 2219c1853e76474799f93fed12cdc12d not found in new config with opid_index 981) from {username='slave'} at 127.0.0.1:60500
I20240501 08:13:09.585165 4644 tablet_replica.cc:330] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: stopping tablet replica
I20240501 08:13:09.585402 4644 raft_consensus.cc:2229] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:09.585577 4644 pending_rounds.cc:62] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: Trying to abort 1 pending ops.
I20240501 08:13:09.585700 4644 pending_rounds.cc:68] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: Aborting op as it isn't in flight: id { term: 2 index: 980 } timestamp: 7022801671475339264 op_type: WRITE_OP write_request { tablet_id: "f4ec90b4ddce408a81f588fa7546e0dd" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\223\001\250m\366P\323\332\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""7d6eb0695bffba02" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022801671369478144 authz_token { token_data: "\010\200\365\307\261\006\"3\n\005slave\022*\n 821c266bb4a949cf878e5f7a6df80abf\020\001\030\001 \001(\001" signature: "<redacted>""pD;\354s\363R\270\310\271\025h\335\211\010\265\364\203\252\3330\2118&T\345\273\223\203\370};\307\3416\344z\260\204\236\311nF\231\377\335ClH\371\002\260>\355X\033\353\221*MTH\313\344" signing_key_seq_num: 0 } } request_id { client_id: "8f1039408fa5432a80634ce45f1a04ab" seq_no: 2932 first_incomplete_seq_no: 2932 attempt_no: 0 }
I20240501 08:13:09.587574 4644 raft_consensus.cc:2258] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:09.589381 4644 ts_tablet_manager.cc:1875] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:09.602214 4644 ts_tablet_manager.cc:1888] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.980
I20240501 08:13:09.602506 4644 log.cc:1196] T f4ec90b4ddce408a81f588fa7546e0dd P 2219c1853e76474799f93fed12cdc12d: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/f4ec90b4ddce408a81f588fa7546e0dd
I20240501 08:13:09.604354 32171 catalog_manager.cc:4830] TS 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223): tablet f4ec90b4ddce408a81f588fa7546e0dd (table rebalance_test_table_2 [id=821c266bb4a949cf878e5f7a6df80abf]) successfully deleted
I20240501 08:13:10.088101 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:10.132820 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:10.138566 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:10.155009 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:10.200335 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:10.323551 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:10.326581 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:10.887651 1292 consensus_queue.cc:237] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1103, Committed index: 1103, Last appended: 3.1103, Last appended by leader: 928, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1104 OBSOLETE_local: false peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } }
I20240501 08:13:10.892808 1555 raft_consensus.cc:1261] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448 [term 3 FOLLOWER]: Refusing update from remote peer 92f76e9a8b5c4bc9b49900f5ca6b4f6e: Log matching property violated. Preceding OpId in replica: term: 3 index: 1103. Preceding OpId from leader: term: 3 index: 1104. (index mismatch)
I20240501 08:13:10.895334 4697 consensus_queue.cc:1035] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1104, Last known committed idx: 1103, Time since last communication: 0.000s
I20240501 08:13:10.900084 4697 raft_consensus.cc:2932] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e [term 3 LEADER]: Committing config change with OpId 3.1104: config changed from index 741 to 1104, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 1104 OBSOLETE_local: false peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:13:10.905774 1555 raft_consensus.cc:2932] T 95a25d38a68a424ebf78f02d2e0750b1 P c5e88f3df73d4d4f996304b6df7e6448 [term 3 FOLLOWER]: Committing config change with OpId 3.1104: config changed from index 741 to 1104, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 1104 OBSOLETE_local: false peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:13:10.919071 32183 catalog_manager.cc:5484] T 95a25d38a68a424ebf78f02d2e0750b1 P 92f76e9a8b5c4bc9b49900f5ca6b4f6e reported cstate change: config changed from index 741 to 1104, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New cstate: current_term: 3 leader_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" committed_config { opid_index: 1104 OBSOLETE_local: false peers { permanent_uuid: "92f76e9a8b5c4bc9b49900f5ca6b4f6e" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 42813 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } }
I20240501 08:13:11.013273 32439 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a25d38a68a424ebf78f02d2e0750b1 with delete_type TABLET_DATA_TOMBSTONED (TS b3e6c8e1eb974b5c9c8b7e109b214576 not found in new config with opid_index 1104) from {username='slave'} at 127.0.0.1:55378
I20240501 08:13:11.017591 4733 tablet_replica.cc:330] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: stopping tablet replica
I20240501 08:13:11.018069 4733 raft_consensus.cc:2229] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:11.024385 4733 raft_consensus.cc:2258] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:11.034231 4733 ts_tablet_manager.cc:1875] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:11.059552 4733 ts_tablet_manager.cc:1888] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1103
I20240501 08:13:11.069036 4733 log.cc:1196] T 95a25d38a68a424ebf78f02d2e0750b1 P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/95a25d38a68a424ebf78f02d2e0750b1
I20240501 08:13:11.074534 32172 catalog_manager.cc:4830] TS b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875): tablet 95a25d38a68a424ebf78f02d2e0750b1 (table rebalance_test_table_3 [id=2cc8097a76e54347ad4bc67576658bc9]) successfully deleted
I20240501 08:13:12.232414 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:12.314159 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:12.376624 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:12.481280 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:12.527426 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:12.630020 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:12.653980 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:13.041352 32456 consensus_queue.cc:237] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 919, Committed index: 919, Last appended: 2.919, Last appended by leader: 814, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 920 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } }
I20240501 08:13:13.046618 1159 raft_consensus.cc:1261] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 2 index: 919. Preceding OpId from leader: term: 2 index: 921. (index mismatch)
I20240501 08:13:13.048365 4575 consensus_queue.cc:1035] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 920, Last known committed idx: 919, Time since last communication: 0.000s
I20240501 08:13:13.061640 4513 raft_consensus.cc:2932] T 789da5b22732463da07c34e8716bbb7e P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Committing config change with OpId 2.920: config changed from index 603 to 920, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 920 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:13:13.064576 1160 raft_consensus.cc:2932] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Committing config change with OpId 2.920: config changed from index 603 to 920, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 920 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:13:13.083388 32183 catalog_manager.cc:5484] T 789da5b22732463da07c34e8716bbb7e P 489550c4b21146918c1c9618be65b4ff reported cstate change: config changed from index 603 to 920, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New cstate: current_term: 2 leader_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" committed_config { opid_index: 920 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } }
I20240501 08:13:13.256072 32306 tablet_service.cc:1514] Processing DeleteTablet for tablet 789da5b22732463da07c34e8716bbb7e with delete_type TABLET_DATA_TOMBSTONED (TS 2219c1853e76474799f93fed12cdc12d not found in new config with opid_index 920) from {username='slave'} at 127.0.0.1:60500
I20240501 08:13:13.278709 4831 tablet_replica.cc:330] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: stopping tablet replica
I20240501 08:13:13.279098 4831 raft_consensus.cc:2229] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:13.279422 4831 raft_consensus.cc:2258] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:13.281123 4831 ts_tablet_manager.cc:1875] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:13.308018 4831 ts_tablet_manager.cc:1888] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.919
I20240501 08:13:13.308280 4831 log.cc:1196] T 789da5b22732463da07c34e8716bbb7e P 2219c1853e76474799f93fed12cdc12d: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/789da5b22732463da07c34e8716bbb7e
I20240501 08:13:13.315305 32171 catalog_manager.cc:4830] TS 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223): tablet 789da5b22732463da07c34e8716bbb7e (table rebalance_test_table_0 [id=56218deb6dfb40f5896115db6d37a8ec]) successfully deleted
I20240501 08:13:14.058208 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:14.084975 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:14.209122 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:14.210769 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:14.247565 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:14.312364 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:14.323114 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 08:13:14.829447 32263 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:39729, user_credentials={real_user=slave}} blocked reactor thread for 47037.5us
I20240501 08:13:15.465613 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:15.467031 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:15.575014 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:15.598940 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:15.662607 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:15.675302 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:15.721311 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:17.960845 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:17.962409 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:18.010133 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:18.118896 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:18.123726 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:18.125916 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:18.126328 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:19.100809 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:19.167186 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:19.168260 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:19.177919 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:19.178107 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:19.209162 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:19.255718 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:20.094411 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:20.094944 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:20.111702 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:20.134653 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:20.135952 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:20.177254 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:20.228788 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bce668f82a0c4f169df6570a5b6df5a5 | 127.25.107.62:43837 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.25.107.20:43851 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
12e44d7017614f88b3205111d85de021 | 127.25.107.3:38795 | HEALTHY | <none> | 5 | 0
2219c1853e76474799f93fed12cdc12d | 127.25.107.1:33223 | HEALTHY | <none> | 4 | 0
37965efda2a34916b49a2357eab43bf3 | 127.25.107.6:44049 | HEALTHY | <none> | 0 | 0
489550c4b21146918c1c9618be65b4ff | 127.25.107.4:39729 | HEALTHY | <none> | 1 | 0
92f76e9a8b5c4bc9b49900f5ca6b4f6e | 127.25.107.5:42813 | HEALTHY | <none> | 1 | 0
b3e6c8e1eb974b5c9c8b7e109b214576 | 127.25.107.2:46875 | HEALTHY | <none> | 3 | 0
c5e88f3df73d4d4f996304b6df7e6448 | 127.25.107.7:46501 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.25.107.1 | experimental | 127.25.107.1:33223
local_ip_for_outbound_sockets | 127.25.107.2 | experimental | 127.25.107.2:46875
local_ip_for_outbound_sockets | 127.25.107.3 | experimental | 127.25.107.3:38795
local_ip_for_outbound_sockets | 127.25.107.4 | experimental | 127.25.107.4:39729
local_ip_for_outbound_sockets | 127.25.107.5 | experimental | 127.25.107.5:42813
local_ip_for_outbound_sockets | 127.25.107.6 | experimental | 127.25.107.6:44049
local_ip_for_outbound_sockets | 127.25.107.7 | experimental | 127.25.107.7:46501
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.25.107.1:33223
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.25.107.2:46875
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.25.107.3:38795
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.25.107.4:39729
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.25.107.5:42813
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.25.107.6:44049
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.25.107.7:46501
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.25.107.20:43851 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 08:13:20.300318 26028 log_verifier.cc:126] Checking tablet 2e794c151e6e40dbb612e64853415d9a
I20240501 08:13:20.672253 26028 log_verifier.cc:177] Verified matching terms for 1138 ops in tablet 2e794c151e6e40dbb612e64853415d9a
I20240501 08:13:20.673609 26028 log_verifier.cc:126] Checking tablet 50c58a563cf94527895a675a1e729fe8
I20240501 08:13:21.097245 26028 log_verifier.cc:177] Verified matching terms for 1194 ops in tablet 50c58a563cf94527895a675a1e729fe8
I20240501 08:13:21.098623 26028 log_verifier.cc:126] Checking tablet 5ba5cd7c680b457d99ee9c6934bbcd0d
I20240501 08:13:21.485570 26028 log_verifier.cc:177] Verified matching terms for 1197 ops in tablet 5ba5cd7c680b457d99ee9c6934bbcd0d
I20240501 08:13:21.487010 26028 log_verifier.cc:126] Checking tablet 5dbd5afc230c41509774812f082f97f0
I20240501 08:13:21.855979 26028 log_verifier.cc:177] Verified matching terms for 1145 ops in tablet 5dbd5afc230c41509774812f082f97f0
I20240501 08:13:21.857270 26028 log_verifier.cc:126] Checking tablet 789da5b22732463da07c34e8716bbb7e
I20240501 08:13:22.192729 26028 log_verifier.cc:177] Verified matching terms for 1018 ops in tablet 789da5b22732463da07c34e8716bbb7e
I20240501 08:13:22.193910 26028 log_verifier.cc:126] Checking tablet 841db634d09c4ef0bdfda733a489af5d
I20240501 08:13:22.544958 26028 log_verifier.cc:177] Verified matching terms for 1067 ops in tablet 841db634d09c4ef0bdfda733a489af5d
I20240501 08:13:22.546242 26028 log_verifier.cc:126] Checking tablet 8bce7a736006455aa01f06e5542022ec
I20240501 08:13:22.898633 26028 log_verifier.cc:177] Verified matching terms for 1082 ops in tablet 8bce7a736006455aa01f06e5542022ec
I20240501 08:13:22.899870 26028 log_verifier.cc:126] Checking tablet 95a25d38a68a424ebf78f02d2e0750b1
I20240501 08:13:23.319182 26028 log_verifier.cc:177] Verified matching terms for 1273 ops in tablet 95a25d38a68a424ebf78f02d2e0750b1
I20240501 08:13:23.320572 26028 log_verifier.cc:126] Checking tablet a50cb92ecb554287977b7a980bd4d2b4
I20240501 08:13:23.356518 26028 log_verifier.cc:177] Verified matching terms for 4 ops in tablet a50cb92ecb554287977b7a980bd4d2b4
I20240501 08:13:23.356613 26028 log_verifier.cc:126] Checking tablet a9d4354546c040d6b971ed6fc3b02c76
I20240501 08:13:23.695741 26028 log_verifier.cc:177] Verified matching terms for 1073 ops in tablet a9d4354546c040d6b971ed6fc3b02c76
I20240501 08:13:23.696979 26028 log_verifier.cc:126] Checking tablet ad5a9e14873742caa50ec838cca1c715
I20240501 08:13:24.092152 26028 log_verifier.cc:177] Verified matching terms for 1158 ops in tablet ad5a9e14873742caa50ec838cca1c715
I20240501 08:13:24.094302 26028 log_verifier.cc:126] Checking tablet adbf537bdcdf4443a94799340aab4205
I20240501 08:13:24.486137 26028 log_verifier.cc:177] Verified matching terms for 1101 ops in tablet adbf537bdcdf4443a94799340aab4205
I20240501 08:13:24.487432 26028 log_verifier.cc:126] Checking tablet b4c8611ac7394bbd96cabee647f5c724
I20240501 08:13:24.865195 26028 log_verifier.cc:177] Verified matching terms for 1136 ops in tablet b4c8611ac7394bbd96cabee647f5c724
I20240501 08:13:24.866497 26028 log_verifier.cc:126] Checking tablet db3c2a3262974c75bba7743192d66d46
I20240501 08:13:25.263916 26028 log_verifier.cc:177] Verified matching terms for 1139 ops in tablet db3c2a3262974c75bba7743192d66d46
I20240501 08:13:25.265554 26028 log_verifier.cc:126] Checking tablet ddaae299e96b412282bec452f9b06fa2
I20240501 08:13:25.653560 26028 log_verifier.cc:177] Verified matching terms for 1193 ops in tablet ddaae299e96b412282bec452f9b06fa2
I20240501 08:13:25.655017 26028 log_verifier.cc:126] Checking tablet f4ec90b4ddce408a81f588fa7546e0dd
I20240501 08:13:26.063064 26028 log_verifier.cc:177] Verified matching terms for 1172 ops in tablet f4ec90b4ddce408a81f588fa7546e0dd
I20240501 08:13:26.078346 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.8:0
--local_ip_for_outbound_sockets=127.25.107.8
--webserver_interface=127.25.107.8
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--builtin_ntp_servers=127.25.107.20:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:13:26.494210 5146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:26.494619 5146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:26.494752 5146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:26.503098 5146 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:13:26.503253 5146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:26.503422 5146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.8
I20240501 08:13:26.513200 5146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.25.107.8
--webserver_port=0
--tserver_master_addrs=127.25.107.62:43837
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.8
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:26.514698 5146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:26.516672 5146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:26.529881 5153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:26.529803 5154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:26.529847 5156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:26.531153 5146 server_base.cc:1034] running on GCE node
I20240501 08:13:26.531788 5146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:26.532810 5146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:26.534035 5146 hybrid_clock.cc:648] HybridClock initialized: now 1714551206533983 us; error 45 us; skew 500 ppm
I20240501 08:13:26.534457 5146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:26.537246 5146 webserver.cc:460] Webserver started at http://127.25.107.8:45303/ using document root <none> and password file <none>
I20240501 08:13:26.538411 5146 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:26.538535 5146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:26.538877 5146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:26.541777 5146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "5341ea9c08294eb1af0f2fd36c1bec7b"
format_stamp: "Formatted at 2024-05-01 08:13:26 on dist-test-slave-dv41"
I20240501 08:13:26.542708 5146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "5341ea9c08294eb1af0f2fd36c1bec7b"
format_stamp: "Formatted at 2024-05-01 08:13:26 on dist-test-slave-dv41"
I20240501 08:13:26.549502 5146 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240501 08:13:26.553617 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:26.555186 5146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:26.555377 5146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "5341ea9c08294eb1af0f2fd36c1bec7b"
format_stamp: "Formatted at 2024-05-01 08:13:26 on dist-test-slave-dv41"
I20240501 08:13:26.555590 5146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:26.588838 5146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:26.589769 5146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:26.590057 5146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:26.591110 5146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:26.592916 5146 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:26.593027 5146 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:26.593123 5146 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:26.593185 5146 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:26.629791 5146 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.8:46277
I20240501 08:13:26.629850 5275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.8:46277 every 8 connection(s)
I20240501 08:13:26.631260 5146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:13:26.638932 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 5146
I20240501 08:13:26.639158 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:13:26.646203 5276 heartbeater.cc:347] Connected to a master server at 127.25.107.62:43837
I20240501 08:13:26.646639 5276 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:26.647451 5276 heartbeater.cc:511] Master 127.25.107.62:43837 requested a full tablet report, sending...
I20240501 08:13:26.649019 32181 ts_manager.cc:194] Registered new tserver with Master: 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8:46277)
I20240501 08:13:26.650027 32181 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.8:58267
I20240501 08:13:27.228248 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:27.237500 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:27.271929 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:27.278561 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:27.304114 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:27.319496 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:27.332160 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:27.347381 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:27.567314 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:27.569658 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:27.573554 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:27.584893 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:27.607481 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:27.608655 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:27.618489 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:27.634208 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:27.653388 5276 heartbeater.cc:503] Master 127.25.107.62:43837 was elected leader, sending a full tablet report...
I20240501 08:13:27.756296 32326 consensus_queue.cc:237] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1136, Committed index: 1136, Last appended: 1.1136, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1137 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:27.759618 1160 raft_consensus.cc:1261] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Refusing update from remote peer 2219c1853e76474799f93fed12cdc12d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1136. Preceding OpId from leader: term: 1 index: 1137. (index mismatch)
I20240501 08:13:27.760715 4485 consensus_queue.cc:1035] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1137, Last known committed idx: 1136, Time since last communication: 0.000s
I20240501 08:13:27.778280 4485 raft_consensus.cc:2932] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Committing config change with OpId 1.1137: config changed from index 496 to 1137, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New config: { opid_index: 1137 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:27.779072 1160 raft_consensus.cc:2932] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Committing config change with OpId 1.1137: config changed from index 496 to 1137, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New config: { opid_index: 1137 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
W20240501 08:13:27.784224 32264 consensus_peers.cc:485] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d -> Peer 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8:46277): Couldn't send request to peer 5341ea9c08294eb1af0f2fd36c1bec7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4c8611ac7394bbd96cabee647f5c724. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:27.788489 32180 catalog_manager.cc:5484] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff reported cstate change: config changed from index 496 to 1137, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "2219c1853e76474799f93fed12cdc12d" committed_config { opid_index: 1137 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:27.829922 32456 consensus_queue.cc:237] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1158, Committed index: 1158, Last appended: 2.1158, Last appended by leader: 549, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:27.832669 1554 raft_consensus.cc:1261] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Refusing update from remote peer b3e6c8e1eb974b5c9c8b7e109b214576: Log matching property violated. Preceding OpId in replica: term: 2 index: 1158. Preceding OpId from leader: term: 2 index: 1159. (index mismatch)
I20240501 08:13:27.833249 5407 consensus_queue.cc:1035] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1159, Last known committed idx: 1158, Time since last communication: 0.000s
I20240501 08:13:27.836270 4513 raft_consensus.cc:2932] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Committing config change with OpId 2.1159: config changed from index 576 to 1159, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New config: { opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:27.836985 1554 raft_consensus.cc:2932] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Committing config change with OpId 2.1159: config changed from index 576 to 1159, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New config: { opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
W20240501 08:13:27.838726 32395 consensus_peers.cc:485] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 -> Peer 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8:46277): Couldn't send request to peer 5341ea9c08294eb1af0f2fd36c1bec7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad5a9e14873742caa50ec838cca1c715. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:27.841583 32185 catalog_manager.cc:5484] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 reported cstate change: config changed from index 576 to 1159, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New cstate: current_term: 2 leader_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" committed_config { opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:27.872730 32588 consensus_queue.cc:237] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1194, Committed index: 1194, Last appended: 1.1194, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1195 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:27.875146 1160 raft_consensus.cc:1261] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 1194. Preceding OpId from leader: term: 1 index: 1195. (index mismatch)
I20240501 08:13:27.875698 5426 consensus_queue.cc:1035] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1195, Last known committed idx: 1194, Time since last communication: 0.000s
I20240501 08:13:27.878365 5162 raft_consensus.cc:2932] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.1195: config changed from index 656 to 1195, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New config: { opid_index: 1195 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:27.879149 1160 raft_consensus.cc:2932] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Committing config change with OpId 1.1195: config changed from index 656 to 1195, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New config: { opid_index: 1195 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
W20240501 08:13:27.881175 32525 consensus_peers.cc:485] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 -> Peer 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8:46277): Couldn't send request to peer 5341ea9c08294eb1af0f2fd36c1bec7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50c58a563cf94527895a675a1e729fe8. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:27.883941 32180 catalog_manager.cc:5484] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index 656 to 1195, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 1195 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:27.915923 32588 consensus_queue.cc:237] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1145, Committed index: 1145, Last appended: 1.1145, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1146 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
W20240501 08:13:27.918107 32525 consensus_peers.cc:485] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 -> Peer 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8:46277): Couldn't send request to peer 5341ea9c08294eb1af0f2fd36c1bec7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5dbd5afc230c41509774812f082f97f0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:27.918107 1420 raft_consensus.cc:1261] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 1145. Preceding OpId from leader: term: 1 index: 1146. (index mismatch)
I20240501 08:13:27.918589 5162 consensus_queue.cc:1035] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1146, Last known committed idx: 1145, Time since last communication: 0.000s
I20240501 08:13:27.921025 5426 raft_consensus.cc:2932] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.1146: config changed from index 529 to 1146, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New config: { opid_index: 1146 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:27.921629 1420 raft_consensus.cc:2932] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Committing config change with OpId 1.1146: config changed from index 529 to 1146, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New config: { opid_index: 1146 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:27.937649 32185 catalog_manager.cc:5484] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index 529 to 1146, VOTER 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 1146 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:28.047992 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:28.059108 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:28.061460 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:28.061901 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:28.077550 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:28.081382 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:28.092945 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:28.149127 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:28.174474 5487 ts_tablet_manager.cc:902] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: Initiating tablet copy from peer 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223)
I20240501 08:13:28.199235 5487 tablet_copy_client.cc:308] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:33223
I20240501 08:13:28.208151 32348 tablet_copy_service.cc:140] P 2219c1853e76474799f93fed12cdc12d: Received BeginTabletCopySession request for tablet b4c8611ac7394bbd96cabee647f5c724 from peer 5341ea9c08294eb1af0f2fd36c1bec7b ({username='slave'} at 127.25.107.8:35125)
I20240501 08:13:28.208330 32348 tablet_copy_service.cc:161] P 2219c1853e76474799f93fed12cdc12d: Beginning new tablet copy session on tablet b4c8611ac7394bbd96cabee647f5c724 from peer 5341ea9c08294eb1af0f2fd36c1bec7b at {username='slave'} at 127.25.107.8:35125: session id = 5341ea9c08294eb1af0f2fd36c1bec7b-b4c8611ac7394bbd96cabee647f5c724
I20240501 08:13:28.211944 32348 tablet_copy_source_session.cc:205] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:28.219480 5487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4c8611ac7394bbd96cabee647f5c724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:28.232064 5487 tablet_copy_client.cc:783] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:28.232477 5487 tablet_copy_client.cc:647] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:28.239053 5487 tablet_copy_client.cc:523] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:28.243243 5487 tablet_bootstrap.cc:492] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap starting.
I20240501 08:13:28.362180 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:28.379544 5514 ts_tablet_manager.cc:902] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b: Initiating tablet copy from peer 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:13:28.380352 5517 ts_tablet_manager.cc:902] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b: Initiating tablet copy from peer b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875)
I20240501 08:13:28.402634 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:28.416993 5517 tablet_copy_client.cc:308] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46875
I20240501 08:13:28.418455 5514 tablet_copy_client.cc:308] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:38795
I20240501 08:13:28.424921 32479 tablet_copy_service.cc:140] P b3e6c8e1eb974b5c9c8b7e109b214576: Received BeginTabletCopySession request for tablet ad5a9e14873742caa50ec838cca1c715 from peer 5341ea9c08294eb1af0f2fd36c1bec7b ({username='slave'} at 127.25.107.8:57287)
I20240501 08:13:28.425108 32479 tablet_copy_service.cc:161] P b3e6c8e1eb974b5c9c8b7e109b214576: Beginning new tablet copy session on tablet ad5a9e14873742caa50ec838cca1c715 from peer 5341ea9c08294eb1af0f2fd36c1bec7b at {username='slave'} at 127.25.107.8:57287: session id = 5341ea9c08294eb1af0f2fd36c1bec7b-ad5a9e14873742caa50ec838cca1c715
I20240501 08:13:28.428041 32479 tablet_copy_source_session.cc:205] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:28.430627 5517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5a9e14873742caa50ec838cca1c715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:28.432104 5487 log.cc:824] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:28.432936 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:28.438113 5517 tablet_copy_client.cc:783] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:28.438632 5517 tablet_copy_client.cc:647] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:28.449856 5517 tablet_copy_client.cc:523] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:28.455523 32609 tablet_copy_service.cc:140] P 12e44d7017614f88b3205111d85de021: Received BeginTabletCopySession request for tablet 50c58a563cf94527895a675a1e729fe8 from peer 5341ea9c08294eb1af0f2fd36c1bec7b ({username='slave'} at 127.25.107.8:36319)
I20240501 08:13:28.455710 32609 tablet_copy_service.cc:161] P 12e44d7017614f88b3205111d85de021: Beginning new tablet copy session on tablet 50c58a563cf94527895a675a1e729fe8 from peer 5341ea9c08294eb1af0f2fd36c1bec7b at {username='slave'} at 127.25.107.8:36319: session id = 5341ea9c08294eb1af0f2fd36c1bec7b-50c58a563cf94527895a675a1e729fe8
I20240501 08:13:28.459080 32609 tablet_copy_source_session.cc:205] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:28.461076 5514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c58a563cf94527895a675a1e729fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:28.461305 5517 tablet_bootstrap.cc:492] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap starting.
I20240501 08:13:28.471330 5514 tablet_copy_client.cc:783] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:28.471741 5514 tablet_copy_client.cc:647] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:28.479957 5514 tablet_copy_client.cc:523] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:28.483952 5514 tablet_bootstrap.cc:492] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap starting.
I20240501 08:13:28.488026 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:28.541703 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:28.544802 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:28.547394 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:28.562996 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:28.574976 5539 ts_tablet_manager.cc:902] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b: Initiating tablet copy from peer 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795)
I20240501 08:13:28.623301 5539 tablet_copy_client.cc:308] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:38795
I20240501 08:13:28.624346 32609 tablet_copy_service.cc:140] P 12e44d7017614f88b3205111d85de021: Received BeginTabletCopySession request for tablet 5dbd5afc230c41509774812f082f97f0 from peer 5341ea9c08294eb1af0f2fd36c1bec7b ({username='slave'} at 127.25.107.8:36319)
I20240501 08:13:28.624543 32609 tablet_copy_service.cc:161] P 12e44d7017614f88b3205111d85de021: Beginning new tablet copy session on tablet 5dbd5afc230c41509774812f082f97f0 from peer 5341ea9c08294eb1af0f2fd36c1bec7b at {username='slave'} at 127.25.107.8:36319: session id = 5341ea9c08294eb1af0f2fd36c1bec7b-5dbd5afc230c41509774812f082f97f0
I20240501 08:13:28.627640 32609 tablet_copy_source_session.cc:205] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:28.630863 5539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbd5afc230c41509774812f082f97f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:28.640672 5539 tablet_copy_client.cc:783] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:28.641085 5539 tablet_copy_client.cc:647] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:28.647332 5539 tablet_copy_client.cc:523] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:28.651491 5539 tablet_bootstrap.cc:492] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap starting.
I20240501 08:13:29.151011 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:29.206519 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:29.238704 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:29.255213 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:29.273675 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:29.298085 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:29.324163 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:29.364639 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:29.942955 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:29.948493 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:29.969067 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:29.972833 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:30.009924 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:30.061122 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:30.146176 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:30.166177 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:13:30.452046 32264 consensus_peers.cc:485] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d -> Peer 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8:46277): Couldn't send request to peer 5341ea9c08294eb1af0f2fd36c1bec7b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 08:13:30.513115 5487 tablet_bootstrap.cc:492] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1137 overwritten=0 applied=1137 ignored=0} inserts{seen=1766 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:30.513978 5487 tablet_bootstrap.cc:492] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap complete.
I20240501 08:13:30.514762 5487 ts_tablet_manager.cc:1367] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: Time spent bootstrapping tablet: real 2.272s user 1.244s sys 0.152s
I20240501 08:13:30.523046 5487 raft_consensus.cc:348] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1137 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:30.523917 5487 raft_consensus.cc:729] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5341ea9c08294eb1af0f2fd36c1bec7b, State: Initialized, Role: FOLLOWER
I20240501 08:13:30.524936 5487 consensus_queue.cc:260] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1137, Last appended: 1.1137, Last appended by leader: 1137, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1137 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:30.533686 5487 ts_tablet_manager.cc:1398] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b: Time spent starting tablet: real 0.019s user 0.008s sys 0.008s
I20240501 08:13:30.535656 32348 tablet_copy_service.cc:342] P 2219c1853e76474799f93fed12cdc12d: Request end of tablet copy session 5341ea9c08294eb1af0f2fd36c1bec7b-b4c8611ac7394bbd96cabee647f5c724 received from {username='slave'} at 127.25.107.8:35125
I20240501 08:13:30.535799 32348 tablet_copy_service.cc:433] P 2219c1853e76474799f93fed12cdc12d: ending tablet copy session 5341ea9c08294eb1af0f2fd36c1bec7b-b4c8611ac7394bbd96cabee647f5c724 on tablet b4c8611ac7394bbd96cabee647f5c724 with peer 5341ea9c08294eb1af0f2fd36c1bec7b
I20240501 08:13:30.682950 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:30.694450 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:30.717705 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:30.719522 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:30.819726 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:13:30.835947 32525 consensus_peers.cc:485] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 -> Peer 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8:46277): Couldn't send request to peer 5341ea9c08294eb1af0f2fd36c1bec7b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 08:13:30.866202 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:30.869370 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:30.871783 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 08:13:30.937064 32395 consensus_peers.cc:485] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 -> Peer 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8:46277): Couldn't send request to peer 5341ea9c08294eb1af0f2fd36c1bec7b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240501 08:13:30.942384 5230 raft_consensus.cc:1203] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1136->[1.1137-1.1137] Dedup: 1.1137->[]
I20240501 08:13:30.993486 5517 tablet_bootstrap.cc:492] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1159 overwritten=0 applied=1159 ignored=0} inserts{seen=1787 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:30.994127 5517 tablet_bootstrap.cc:492] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap complete.
I20240501 08:13:30.994715 5517 ts_tablet_manager.cc:1367] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b: Time spent bootstrapping tablet: real 2.534s user 1.280s sys 0.125s
I20240501 08:13:30.996515 5517 raft_consensus.cc:348] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:30.996834 5517 raft_consensus.cc:729] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5341ea9c08294eb1af0f2fd36c1bec7b, State: Initialized, Role: FOLLOWER
I20240501 08:13:30.997207 5517 consensus_queue.cc:260] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1159, Last appended: 2.1159, Last appended by leader: 1159, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:30.998703 5517 ts_tablet_manager.cc:1398] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:13:30.999476 32479 tablet_copy_service.cc:342] P b3e6c8e1eb974b5c9c8b7e109b214576: Request end of tablet copy session 5341ea9c08294eb1af0f2fd36c1bec7b-ad5a9e14873742caa50ec838cca1c715 received from {username='slave'} at 127.25.107.8:57287
I20240501 08:13:30.999610 32479 tablet_copy_service.cc:433] P b3e6c8e1eb974b5c9c8b7e109b214576: ending tablet copy session 5341ea9c08294eb1af0f2fd36c1bec7b-ad5a9e14873742caa50ec838cca1c715 on tablet ad5a9e14873742caa50ec838cca1c715 with peer 5341ea9c08294eb1af0f2fd36c1bec7b
I20240501 08:13:31.010665 5539 tablet_bootstrap.cc:492] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1146 overwritten=0 applied=1146 ignored=0} inserts{seen=1829 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:31.011274 5539 tablet_bootstrap.cc:492] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap complete.
I20240501 08:13:31.011781 5539 ts_tablet_manager.cc:1367] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b: Time spent bootstrapping tablet: real 2.360s user 1.266s sys 0.134s
I20240501 08:13:31.012562 5539 raft_consensus.cc:348] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1146 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:31.012835 5539 raft_consensus.cc:729] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5341ea9c08294eb1af0f2fd36c1bec7b, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.013172 5539 consensus_queue.cc:260] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1146, Last appended: 1.1146, Last appended by leader: 1146, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1146 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:31.014168 5539 ts_tablet_manager.cc:1398] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:31.015259 32609 tablet_copy_service.cc:342] P 12e44d7017614f88b3205111d85de021: Request end of tablet copy session 5341ea9c08294eb1af0f2fd36c1bec7b-5dbd5afc230c41509774812f082f97f0 received from {username='slave'} at 127.25.107.8:36319
I20240501 08:13:31.015414 32609 tablet_copy_service.cc:433] P 12e44d7017614f88b3205111d85de021: ending tablet copy session 5341ea9c08294eb1af0f2fd36c1bec7b-5dbd5afc230c41509774812f082f97f0 on tablet 5dbd5afc230c41509774812f082f97f0 with peer 5341ea9c08294eb1af0f2fd36c1bec7b
I20240501 08:13:31.079910 5514 tablet_bootstrap.cc:492] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1195 overwritten=0 applied=1195 ignored=0} inserts{seen=1824 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:31.080377 5514 tablet_bootstrap.cc:492] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b: Bootstrap complete.
I20240501 08:13:31.080735 5514 ts_tablet_manager.cc:1367] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b: Time spent bootstrapping tablet: real 2.597s user 1.298s sys 0.127s
I20240501 08:13:31.081461 5514 raft_consensus.cc:348] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1195 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:31.081737 5514 raft_consensus.cc:729] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5341ea9c08294eb1af0f2fd36c1bec7b, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.081981 5514 consensus_queue.cc:260] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1195, Last appended: 1.1195, Last appended by leader: 1195, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1195 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:31.082530 5514 ts_tablet_manager.cc:1398] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:13:31.083388 32609 tablet_copy_service.cc:342] P 12e44d7017614f88b3205111d85de021: Request end of tablet copy session 5341ea9c08294eb1af0f2fd36c1bec7b-50c58a563cf94527895a675a1e729fe8 received from {username='slave'} at 127.25.107.8:36319
I20240501 08:13:31.083515 32609 tablet_copy_service.cc:433] P 12e44d7017614f88b3205111d85de021: ending tablet copy session 5341ea9c08294eb1af0f2fd36c1bec7b-50c58a563cf94527895a675a1e729fe8 on tablet 50c58a563cf94527895a675a1e729fe8 with peer 5341ea9c08294eb1af0f2fd36c1bec7b
I20240501 08:13:31.145104 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:31.158671 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:31.170871 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:31.172856 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:31.180933 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:31.207198 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:31.211374 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:31.215077 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:31.238848 5230 raft_consensus.cc:1203] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1145->[1.1146-1.1146] Dedup: 1.1146->[]
I20240501 08:13:31.401924 5230 raft_consensus.cc:1203] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1158->[2.1159-2.1159] Dedup: 2.1159->[]
I20240501 08:13:31.402365 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:31.417675 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:31.420516 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:31.424363 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:31.424852 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:31.436429 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:31.441148 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:31.449714 5230 raft_consensus.cc:1203] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1194->[1.1195-1.1195] Dedup: 1.1195->[]
I20240501 08:13:31.476488 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:31.847676 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:31.851366 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:31.863068 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:31.867432 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:31.889323 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:31.906312 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:31.982843 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:31.983017 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:32.042199 32588 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5dbd5afc230c41509774812f082f97f0"
dest_uuid: "12e44d7017614f88b3205111d85de021"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35906
I20240501 08:13:32.042426 32588 raft_consensus.cc:595] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:13:32.144954 32569 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:13:32.146422 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:32.150537 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:32.152388 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:32.180876 5211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:32.190367 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:32.233142 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:32.245405 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:32.356761 5844 raft_consensus.cc:979] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: : Instructing follower 5341ea9c08294eb1af0f2fd36c1bec7b to start an election
I20240501 08:13:32.356952 5844 raft_consensus.cc:1067] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Signalling peer 5341ea9c08294eb1af0f2fd36c1bec7b to start an election
I20240501 08:13:32.357879 5231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5dbd5afc230c41509774812f082f97f0"
dest_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b"
from {username='slave'} at 127.25.107.3:46219
I20240501 08:13:32.358208 5231 raft_consensus.cc:482] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:32.358448 5231 raft_consensus.cc:3037] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:32.361037 5231 raft_consensus.cc:504] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1146 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:32.362632 5231 leader_election.cc:288] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [CANDIDATE]: Term 2 election: Requested vote from peers 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795), 37965efda2a34916b49a2357eab43bf3 (127.25.107.6:44049)
I20240501 08:13:32.363086 32588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbd5afc230c41509774812f082f97f0" candidate_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1146 } } ignore_live_leader: true dest_uuid: "12e44d7017614f88b3205111d85de021"
I20240501 08:13:32.363334 32588 raft_consensus.cc:3032] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:32.363420 32588 raft_consensus.cc:729] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12e44d7017614f88b3205111d85de021, State: Running, Role: LEADER
I20240501 08:13:32.363708 32588 consensus_queue.cc:260] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1146, Committed index: 1146, Last appended: 1.1146, Last appended by leader: 1146, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1146 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:32.364099 32588 raft_consensus.cc:3037] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:32.367216 32588 raft_consensus.cc:2454] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5341ea9c08294eb1af0f2fd36c1bec7b in term 2.
I20240501 08:13:32.367992 5166 leader_election.cc:302] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e44d7017614f88b3205111d85de021, 5341ea9c08294eb1af0f2fd36c1bec7b; no voters:
I20240501 08:13:32.371778 1423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbd5afc230c41509774812f082f97f0" candidate_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1146 } } ignore_live_leader: true dest_uuid: "37965efda2a34916b49a2357eab43bf3"
I20240501 08:13:32.372165 1423 raft_consensus.cc:3037] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:32.375628 1423 raft_consensus.cc:2454] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5341ea9c08294eb1af0f2fd36c1bec7b in term 2.
I20240501 08:13:32.377437 5851 raft_consensus.cc:2781] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:32.379056 5851 raft_consensus.cc:686] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 LEADER]: Becoming Leader. State: Replica: 5341ea9c08294eb1af0f2fd36c1bec7b, State: Running, Role: LEADER
I20240501 08:13:32.379750 5851 consensus_queue.cc:237] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1146, Committed index: 1146, Last appended: 1.1146, Last appended by leader: 1146, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1146 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:32.389572 32185 catalog_manager.cc:5484] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b reported cstate change: term changed from 1 to 2, leader changed from 12e44d7017614f88b3205111d85de021 (127.25.107.3) to 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8). New cstate: current_term: 2 leader_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" committed_config { opid_index: 1146 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:32.435338 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:32.438398 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:32.439815 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:32.449110 5211 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:32.463995 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:32.469731 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:32.470723 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:32.477044 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:32.590200 32588 consensus_queue.cc:237] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1195, Committed index: 1195, Last appended: 1.1195, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1196 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:32.592468 5231 raft_consensus.cc:1261] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Refusing update from remote peer 12e44d7017614f88b3205111d85de021: Log matching property violated. Preceding OpId in replica: term: 1 index: 1195. Preceding OpId from leader: term: 1 index: 1196. (index mismatch)
I20240501 08:13:32.593140 5844 consensus_queue.cc:1035] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1196, Last known committed idx: 1195, Time since last communication: 0.000s
I20240501 08:13:32.598093 5756 raft_consensus.cc:2932] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 [term 1 LEADER]: Committing config change with OpId 1.1196: config changed from index 1195 to 1196, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) evicted. New config: { opid_index: 1196 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:32.599061 5231 raft_consensus.cc:2932] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Committing config change with OpId 1.1196: config changed from index 1195 to 1196, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) evicted. New config: { opid_index: 1196 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:32.602799 32182 catalog_manager.cc:5484] T 50c58a563cf94527895a675a1e729fe8 P 12e44d7017614f88b3205111d85de021 reported cstate change: config changed from index 1195 to 1196, VOTER 489550c4b21146918c1c9618be65b4ff (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "12e44d7017614f88b3205111d85de021" committed_config { opid_index: 1196 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:32.614203 1142 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c58a563cf94527895a675a1e729fe8 with delete_type TABLET_DATA_TOMBSTONED (TS 489550c4b21146918c1c9618be65b4ff not found in new config with opid_index 1196) from {username='slave'} at 127.0.0.1:33018
I20240501 08:13:32.622808 5904 tablet_replica.cc:330] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: stopping tablet replica
I20240501 08:13:32.623293 5904 raft_consensus.cc:2229] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:32.623644 5904 raft_consensus.cc:2258] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:32.626066 5904 ts_tablet_manager.cc:1875] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:32.632553 5904 ts_tablet_manager.cc:1888] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1195
I20240501 08:13:32.632854 5904 log.cc:1196] T 50c58a563cf94527895a675a1e729fe8 P 489550c4b21146918c1c9618be65b4ff: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/50c58a563cf94527895a675a1e729fe8
I20240501 08:13:32.633999 32173 catalog_manager.cc:4830] TS 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729): tablet 50c58a563cf94527895a675a1e729fe8 (table rebalance_test_table_3 [id=2cc8097a76e54347ad4bc67576658bc9]) successfully deleted
I20240501 08:13:32.736014 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:32.744100 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:32.760076 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:32.760524 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:32.809818 5211 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:32.822628 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:32.847322 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:32.849341 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:32.856140 1423 raft_consensus.cc:1261] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 2 FOLLOWER]: Refusing update from remote peer 5341ea9c08294eb1af0f2fd36c1bec7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1146. Preceding OpId from leader: term: 2 index: 1147. (index mismatch)
I20240501 08:13:32.861671 5851 consensus_queue.cc:1035] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1147, Last known committed idx: 1146, Time since last communication: 0.000s
I20240501 08:13:32.891247 32588 raft_consensus.cc:1261] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Refusing update from remote peer 5341ea9c08294eb1af0f2fd36c1bec7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1146. Preceding OpId from leader: term: 2 index: 1147. (index mismatch)
I20240501 08:13:32.892148 5856 consensus_queue.cc:1035] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1147, Last known committed idx: 1146, Time since last communication: 0.000s
I20240501 08:13:32.929247 32326 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4c8611ac7394bbd96cabee647f5c724"
dest_uuid: "2219c1853e76474799f93fed12cdc12d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59508
I20240501 08:13:32.929481 32326 raft_consensus.cc:595] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Received request to transfer leadership
I20240501 08:13:33.043059 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:33.044907 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:33.046466 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:33.046937 32306 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:33.081923 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:33.094717 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:33.103432 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:33.128055 5211 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:33.167495 5984 raft_consensus.cc:979] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: : Instructing follower 5341ea9c08294eb1af0f2fd36c1bec7b to start an election
I20240501 08:13:33.167661 5984 raft_consensus.cc:1067] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Signalling peer 5341ea9c08294eb1af0f2fd36c1bec7b to start an election
I20240501 08:13:33.168776 5231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4c8611ac7394bbd96cabee647f5c724"
dest_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b"
from {username='slave'} at 127.25.107.1:49399
I20240501 08:13:33.169039 5231 raft_consensus.cc:482] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:33.169157 5231 raft_consensus.cc:3037] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:33.171954 5231 raft_consensus.cc:504] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1137 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:33.193255 5231 leader_election.cc:288] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [CANDIDATE]: Term 2 election: Requested vote from peers 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223), 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729)
I20240501 08:13:33.193884 32326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c8611ac7394bbd96cabee647f5c724" candidate_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1137 } } ignore_live_leader: true dest_uuid: "2219c1853e76474799f93fed12cdc12d"
I20240501 08:13:33.194118 32326 raft_consensus.cc:3032] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:33.194185 32326 raft_consensus.cc:729] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2219c1853e76474799f93fed12cdc12d, State: Running, Role: LEADER
I20240501 08:13:33.194468 32326 consensus_queue.cc:260] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1137, Committed index: 1137, Last appended: 1.1137, Last appended by leader: 1137, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1137 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:33.194773 32326 raft_consensus.cc:3037] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:33.198401 32326 raft_consensus.cc:2454] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5341ea9c08294eb1af0f2fd36c1bec7b in term 2.
I20240501 08:13:33.198963 5164 leader_election.cc:302] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2219c1853e76474799f93fed12cdc12d, 5341ea9c08294eb1af0f2fd36c1bec7b; no voters:
I20240501 08:13:33.199291 5851 raft_consensus.cc:2781] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:33.199446 5851 raft_consensus.cc:686] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 LEADER]: Becoming Leader. State: Replica: 5341ea9c08294eb1af0f2fd36c1bec7b, State: Running, Role: LEADER
I20240501 08:13:33.199710 5851 consensus_queue.cc:237] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1137, Committed index: 1137, Last appended: 1.1137, Last appended by leader: 1137, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1137 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:33.202772 32180 catalog_manager.cc:5484] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b reported cstate change: term changed from 1 to 2, leader changed from 2219c1853e76474799f93fed12cdc12d (127.25.107.1) to 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8). New cstate: current_term: 2 leader_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" committed_config { opid_index: 1137 OBSOLETE_local: false peers { permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 } } peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:33.213733 1160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4c8611ac7394bbd96cabee647f5c724" candidate_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1137 } } ignore_live_leader: true dest_uuid: "489550c4b21146918c1c9618be65b4ff"
I20240501 08:13:33.213989 1160 raft_consensus.cc:3037] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:33.216935 1160 raft_consensus.cc:2454] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5341ea9c08294eb1af0f2fd36c1bec7b in term 2.
I20240501 08:13:33.372619 32306 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:33.373984 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:33.373972 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:33.394647 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:33.394866 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:33.417002 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:33.438246 5211 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:33.454538 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:13:33.474828 1226 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:46501, user_credentials={real_user=slave}} blocked reactor thread for 49383.8us
I20240501 08:13:33.541354 32456 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ad5a9e14873742caa50ec838cca1c715"
dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60956
I20240501 08:13:33.541582 32456 raft_consensus.cc:595] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Received request to transfer leadership
I20240501 08:13:33.642179 1160 raft_consensus.cc:1261] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Refusing update from remote peer 5341ea9c08294eb1af0f2fd36c1bec7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1137. Preceding OpId from leader: term: 2 index: 1138. (index mismatch)
I20240501 08:13:33.643293 5851 consensus_queue.cc:1035] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1138, Last known committed idx: 1137, Time since last communication: 0.000s
I20240501 08:13:33.648419 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:33.653657 32326 raft_consensus.cc:1261] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Refusing update from remote peer 5341ea9c08294eb1af0f2fd36c1bec7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1137. Preceding OpId from leader: term: 2 index: 1138. (index mismatch)
I20240501 08:13:33.658625 5856 consensus_queue.cc:1035] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2219c1853e76474799f93fed12cdc12d" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1138, Last known committed idx: 1137, Time since last communication: 0.000s
I20240501 08:13:33.660794 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:33.682334 32306 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:33.697436 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:33.742910 5211 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:33.750957 5407 raft_consensus.cc:979] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: : Instructing follower 5341ea9c08294eb1af0f2fd36c1bec7b to start an election
I20240501 08:13:33.756799 6083 raft_consensus.cc:1067] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Signalling peer 5341ea9c08294eb1af0f2fd36c1bec7b to start an election
I20240501 08:13:33.762210 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:33.771525 5231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ad5a9e14873742caa50ec838cca1c715"
dest_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b"
from {username='slave'} at 127.25.107.2:43025
I20240501 08:13:33.771840 5231 raft_consensus.cc:482] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:33.771987 5231 raft_consensus.cc:3037] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:33.779443 5231 raft_consensus.cc:504] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:33.781569 32439 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:33.783016 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5a9e14873742caa50ec838cca1c715" candidate_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1159 } } ignore_live_leader: true dest_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576"
I20240501 08:13:33.783252 32459 raft_consensus.cc:3032] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:33.783324 32459 raft_consensus.cc:729] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b3e6c8e1eb974b5c9c8b7e109b214576, State: Running, Role: LEADER
I20240501 08:13:33.783572 32459 consensus_queue.cc:260] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1159, Committed index: 1159, Last appended: 2.1159, Last appended by leader: 1159, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:33.783903 32459 raft_consensus.cc:3037] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:33.789156 32459 raft_consensus.cc:2454] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5341ea9c08294eb1af0f2fd36c1bec7b in term 3.
I20240501 08:13:33.789768 5165 leader_election.cc:302] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5341ea9c08294eb1af0f2fd36c1bec7b, b3e6c8e1eb974b5c9c8b7e109b214576; no voters:
I20240501 08:13:33.790133 5856 raft_consensus.cc:2781] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:33.790242 5856 raft_consensus.cc:686] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 3 LEADER]: Becoming Leader. State: Replica: 5341ea9c08294eb1af0f2fd36c1bec7b, State: Running, Role: LEADER
I20240501 08:13:33.790844 5231 leader_election.cc:288] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [CANDIDATE]: Term 3 election: Requested vote from peers b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875), c5e88f3df73d4d4f996304b6df7e6448 (127.25.107.7:46501)
I20240501 08:13:33.790728 5856 consensus_queue.cc:237] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1159, Committed index: 1159, Last appended: 2.1159, Last appended by leader: 1159, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:33.795626 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:33.813277 32185 catalog_manager.cc:5484] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b reported cstate change: term changed from 2 to 3, leader changed from b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) to 5341ea9c08294eb1af0f2fd36c1bec7b (127.25.107.8). New cstate: current_term: 3 leader_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" committed_config { opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 } } peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:33.815148 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5a9e14873742caa50ec838cca1c715" candidate_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1159 } } ignore_live_leader: true dest_uuid: "c5e88f3df73d4d4f996304b6df7e6448"
I20240501 08:13:33.815428 1554 raft_consensus.cc:3037] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:33.819308 1554 raft_consensus.cc:2454] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5341ea9c08294eb1af0f2fd36c1bec7b in term 3.
I20240501 08:13:34.209892 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:34.236886 32306 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:34.280341 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:34.285655 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:34.298936 5211 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:34.313280 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:34.331727 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:34.352823 32459 raft_consensus.cc:1261] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 3 FOLLOWER]: Refusing update from remote peer 5341ea9c08294eb1af0f2fd36c1bec7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1159. Preceding OpId from leader: term: 3 index: 1160. (index mismatch)
I20240501 08:13:34.353699 5851 consensus_queue.cc:1035] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e6c8e1eb974b5c9c8b7e109b214576" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1160, Last known committed idx: 1159, Time since last communication: 0.000s
I20240501 08:13:34.368537 1554 raft_consensus.cc:1261] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 3 FOLLOWER]: Refusing update from remote peer 5341ea9c08294eb1af0f2fd36c1bec7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1159. Preceding OpId from leader: term: 3 index: 1160. (index mismatch)
I20240501 08:13:34.369516 5856 consensus_queue.cc:1035] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1160, Last known committed idx: 1159, Time since last communication: 0.000s
I20240501 08:13:34.376444 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:34.451815 5231 consensus_queue.cc:237] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1147, Committed index: 1147, Last appended: 2.1147, Last appended by leader: 1146, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1148 OBSOLETE_local: false peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:34.454051 1423 raft_consensus.cc:1261] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 2 FOLLOWER]: Refusing update from remote peer 5341ea9c08294eb1af0f2fd36c1bec7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1147. Preceding OpId from leader: term: 2 index: 1148. (index mismatch)
I20240501 08:13:34.454632 5851 consensus_queue.cc:1035] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1148, Last known committed idx: 1147, Time since last communication: 0.000s
I20240501 08:13:34.457158 5856 raft_consensus.cc:2932] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 LEADER]: Committing config change with OpId 2.1148: config changed from index 1146 to 1148, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 1148 OBSOLETE_local: false peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:34.457805 1423 raft_consensus.cc:2932] T 5dbd5afc230c41509774812f082f97f0 P 37965efda2a34916b49a2357eab43bf3 [term 2 FOLLOWER]: Committing config change with OpId 2.1148: config changed from index 1146 to 1148, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New config: { opid_index: 1148 OBSOLETE_local: false peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:34.463092 32182 catalog_manager.cc:5484] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b reported cstate change: config changed from index 1146 to 1148, VOTER 12e44d7017614f88b3205111d85de021 (127.25.107.3) evicted. New cstate: current_term: 2 leader_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" committed_config { opid_index: 1148 OBSOLETE_local: false peers { permanent_uuid: "37965efda2a34916b49a2357eab43bf3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 44049 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:34.474330 32569 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbd5afc230c41509774812f082f97f0 with delete_type TABLET_DATA_TOMBSTONED (TS 12e44d7017614f88b3205111d85de021 not found in new config with opid_index 1148) from {username='slave'} at 127.0.0.1:39982
I20240501 08:13:34.475015 6176 tablet_replica.cc:330] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: stopping tablet replica
I20240501 08:13:34.475239 6176 raft_consensus.cc:2229] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:34.475541 6176 raft_consensus.cc:2258] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:34.476970 6176 ts_tablet_manager.cc:1875] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:34.482841 6176 ts_tablet_manager.cc:1888] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1147
I20240501 08:13:34.482981 6176 log.cc:1196] T 5dbd5afc230c41509774812f082f97f0 P 12e44d7017614f88b3205111d85de021: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/5dbd5afc230c41509774812f082f97f0
I20240501 08:13:34.483762 32173 catalog_manager.cc:4830] TS 12e44d7017614f88b3205111d85de021 (127.25.107.3:38795): tablet 5dbd5afc230c41509774812f082f97f0 (table rebalance_test_table_2 [id=821c266bb4a949cf878e5f7a6df80abf]) successfully deleted
I20240501 08:13:34.591264 32306 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:34.597594 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:34.600023 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:34.601241 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:34.619024 5211 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:34.647207 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:34.664831 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:34.674098 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:34.785820 5231 consensus_queue.cc:237] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1138, Committed index: 1138, Last appended: 2.1138, Last appended by leader: 1137, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1139 OBSOLETE_local: false peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:34.787773 1160 raft_consensus.cc:1261] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Refusing update from remote peer 5341ea9c08294eb1af0f2fd36c1bec7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1138. Preceding OpId from leader: term: 2 index: 1139. (index mismatch)
I20240501 08:13:34.788386 5851 consensus_queue.cc:1035] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1139, Last known committed idx: 1138, Time since last communication: 0.000s
I20240501 08:13:34.791098 5856 raft_consensus.cc:2932] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 2 LEADER]: Committing config change with OpId 2.1139: config changed from index 1137 to 1139, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 1139 OBSOLETE_local: false peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:34.791930 1160 raft_consensus.cc:2932] T b4c8611ac7394bbd96cabee647f5c724 P 489550c4b21146918c1c9618be65b4ff [term 2 FOLLOWER]: Committing config change with OpId 2.1139: config changed from index 1137 to 1139, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New config: { opid_index: 1139 OBSOLETE_local: false peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:34.797019 32180 catalog_manager.cc:5484] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b reported cstate change: config changed from index 1137 to 1139, VOTER 2219c1853e76474799f93fed12cdc12d (127.25.107.1) evicted. New cstate: current_term: 2 leader_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" committed_config { opid_index: 1139 OBSOLETE_local: false peers { permanent_uuid: "489550c4b21146918c1c9618be65b4ff" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 39729 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:34.801374 32306 tablet_service.cc:1514] Processing DeleteTablet for tablet b4c8611ac7394bbd96cabee647f5c724 with delete_type TABLET_DATA_TOMBSTONED (TS 2219c1853e76474799f93fed12cdc12d not found in new config with opid_index 1139) from {username='slave'} at 127.0.0.1:60500
I20240501 08:13:34.802631 6233 tablet_replica.cc:330] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: stopping tablet replica
I20240501 08:13:34.802906 6233 raft_consensus.cc:2229] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:34.803102 6233 raft_consensus.cc:2258] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:34.804430 6233 ts_tablet_manager.cc:1875] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:34.811442 6233 ts_tablet_manager.cc:1888] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1138
I20240501 08:13:34.811631 6233 log.cc:1196] T b4c8611ac7394bbd96cabee647f5c724 P 2219c1853e76474799f93fed12cdc12d: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/b4c8611ac7394bbd96cabee647f5c724
I20240501 08:13:34.812614 32171 catalog_manager.cc:4830] TS 2219c1853e76474799f93fed12cdc12d (127.25.107.1:33223): tablet b4c8611ac7394bbd96cabee647f5c724 (table rebalance_test_table_4 [id=e2d2d2401bec456884eaceee2e5edf67]) successfully deleted
I20240501 08:13:34.918059 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:34.919042 32306 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:34.931998 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:34.938586 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:34.944865 5211 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:34.954628 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:34.957721 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:34.972921 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:35.075433 5231 consensus_queue.cc:237] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1160, Committed index: 1160, Last appended: 3.1160, Last appended by leader: 1159, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1161 OBSOLETE_local: false peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:35.077065 1554 raft_consensus.cc:1261] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 3 FOLLOWER]: Refusing update from remote peer 5341ea9c08294eb1af0f2fd36c1bec7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 1160. Preceding OpId from leader: term: 3 index: 1161. (index mismatch)
I20240501 08:13:35.077670 5856 consensus_queue.cc:1035] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1161, Last known committed idx: 1160, Time since last communication: 0.000s
I20240501 08:13:35.080219 5851 raft_consensus.cc:2932] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 3 LEADER]: Committing config change with OpId 3.1161: config changed from index 1159 to 1161, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 1161 OBSOLETE_local: false peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:35.080915 1554 raft_consensus.cc:2932] T ad5a9e14873742caa50ec838cca1c715 P c5e88f3df73d4d4f996304b6df7e6448 [term 3 FOLLOWER]: Committing config change with OpId 3.1161: config changed from index 1159 to 1161, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New config: { opid_index: 1161 OBSOLETE_local: false peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:35.085933 32180 catalog_manager.cc:5484] T ad5a9e14873742caa50ec838cca1c715 P 5341ea9c08294eb1af0f2fd36c1bec7b reported cstate change: config changed from index 1159 to 1161, VOTER b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2) evicted. New cstate: current_term: 3 leader_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" committed_config { opid_index: 1161 OBSOLETE_local: false peers { permanent_uuid: "c5e88f3df73d4d4f996304b6df7e6448" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 46501 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } } }
I20240501 08:13:35.090884 32439 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5a9e14873742caa50ec838cca1c715 with delete_type TABLET_DATA_TOMBSTONED (TS b3e6c8e1eb974b5c9c8b7e109b214576 not found in new config with opid_index 1161) from {username='slave'} at 127.0.0.1:55378
I20240501 08:13:35.091526 6287 tablet_replica.cc:330] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: stopping tablet replica
I20240501 08:13:35.091795 6287 raft_consensus.cc:2229] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:35.091992 6287 raft_consensus.cc:2258] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:35.093506 6287 ts_tablet_manager.cc:1875] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:35.099118 6287 ts_tablet_manager.cc:1888] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1160
I20240501 08:13:35.099287 6287 log.cc:1196] T ad5a9e14873742caa50ec838cca1c715 P b3e6c8e1eb974b5c9c8b7e109b214576: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/ad5a9e14873742caa50ec838cca1c715
I20240501 08:13:35.100208 32172 catalog_manager.cc:4830] TS b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875): tablet ad5a9e14873742caa50ec838cca1c715 (table rebalance_test_table_1 [id=3910e4f153974479aa62dc08cd2c1ac3]) successfully deleted
I20240501 08:13:35.262521 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:35.264155 32306 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:35.265623 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:35.275818 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:35.279726 5211 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:35.313537 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:35.328981 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:35.334887 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:35.520474 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:35.529384 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:35.543325 32306 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:35.548321 5211 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:35.570212 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:35.570314 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:35.586843 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:35.588578 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:36.071522 32306 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:36.109948 1142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:36.116781 1403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:13:36.146054 5211 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:13:36.146648 32569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:13:36.208923 1535 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:13:36.248909 1273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:13:36.256258 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bce668f82a0c4f169df6570a5b6df5a5 | 127.25.107.62:43837 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.25.107.20:43851 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
12e44d7017614f88b3205111d85de021 | 127.25.107.3:38795 | HEALTHY | <none> | 4 | 0
2219c1853e76474799f93fed12cdc12d | 127.25.107.1:33223 | HEALTHY | <none> | 3 | 0
37965efda2a34916b49a2357eab43bf3 | 127.25.107.6:44049 | HEALTHY | <none> | 0 | 0
489550c4b21146918c1c9618be65b4ff | 127.25.107.4:39729 | HEALTHY | <none> | 1 | 0
5341ea9c08294eb1af0f2fd36c1bec7b | 127.25.107.8:46277 | HEALTHY | <none> | 3 | 0
92f76e9a8b5c4bc9b49900f5ca6b4f6e | 127.25.107.5:42813 | HEALTHY | <none> | 1 | 0
b3e6c8e1eb974b5c9c8b7e109b214576 | 127.25.107.2:46875 | HEALTHY | <none> | 2 | 0
c5e88f3df73d4d4f996304b6df7e6448 | 127.25.107.7:46501 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.25.107.1 | experimental | 127.25.107.1:33223
local_ip_for_outbound_sockets | 127.25.107.2 | experimental | 127.25.107.2:46875
local_ip_for_outbound_sockets | 127.25.107.3 | experimental | 127.25.107.3:38795
local_ip_for_outbound_sockets | 127.25.107.4 | experimental | 127.25.107.4:39729
local_ip_for_outbound_sockets | 127.25.107.5 | experimental | 127.25.107.5:42813
local_ip_for_outbound_sockets | 127.25.107.6 | experimental | 127.25.107.6:44049
local_ip_for_outbound_sockets | 127.25.107.7 | experimental | 127.25.107.7:46501
local_ip_for_outbound_sockets | 127.25.107.8 | experimental | 127.25.107.8:46277
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.25.107.1:33223
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.25.107.2:46875
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.25.107.3:38795
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.25.107.4:39729
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.25.107.5:42813
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.25.107.6:44049
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.25.107.7:46501
server_dump_info_path | /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.25.107.8:46277
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.25.107.20:43851 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 08:13:36.310797 26028 log_verifier.cc:126] Checking tablet 2e794c151e6e40dbb612e64853415d9a
I20240501 08:13:36.729175 26028 log_verifier.cc:177] Verified matching terms for 1138 ops in tablet 2e794c151e6e40dbb612e64853415d9a
I20240501 08:13:36.730628 26028 log_verifier.cc:126] Checking tablet 50c58a563cf94527895a675a1e729fe8
I20240501 08:13:37.127663 26028 log_verifier.cc:177] Verified matching terms for 1196 ops in tablet 50c58a563cf94527895a675a1e729fe8
I20240501 08:13:37.129038 26028 log_verifier.cc:126] Checking tablet 5ba5cd7c680b457d99ee9c6934bbcd0d
I20240501 08:13:37.546770 26028 log_verifier.cc:177] Verified matching terms for 1197 ops in tablet 5ba5cd7c680b457d99ee9c6934bbcd0d
I20240501 08:13:37.548097 26028 log_verifier.cc:126] Checking tablet 5dbd5afc230c41509774812f082f97f0
I20240501 08:13:37.911615 26028 log_verifier.cc:177] Verified matching terms for 1148 ops in tablet 5dbd5afc230c41509774812f082f97f0
I20240501 08:13:37.912923 26028 log_verifier.cc:126] Checking tablet 789da5b22732463da07c34e8716bbb7e
I20240501 08:13:38.268957 26028 log_verifier.cc:177] Verified matching terms for 1018 ops in tablet 789da5b22732463da07c34e8716bbb7e
I20240501 08:13:38.270179 26028 log_verifier.cc:126] Checking tablet 841db634d09c4ef0bdfda733a489af5d
I20240501 08:13:38.575507 26028 log_verifier.cc:177] Verified matching terms for 1067 ops in tablet 841db634d09c4ef0bdfda733a489af5d
I20240501 08:13:38.576715 26028 log_verifier.cc:126] Checking tablet 8bce7a736006455aa01f06e5542022ec
I20240501 08:13:38.934675 26028 log_verifier.cc:177] Verified matching terms for 1082 ops in tablet 8bce7a736006455aa01f06e5542022ec
I20240501 08:13:38.935936 26028 log_verifier.cc:126] Checking tablet 95a25d38a68a424ebf78f02d2e0750b1
I20240501 08:13:39.304910 26028 log_verifier.cc:177] Verified matching terms for 1273 ops in tablet 95a25d38a68a424ebf78f02d2e0750b1
I20240501 08:13:39.306401 26028 log_verifier.cc:126] Checking tablet a50cb92ecb554287977b7a980bd4d2b4
I20240501 08:13:39.317312 26028 log_verifier.cc:177] Verified matching terms for 4 ops in tablet a50cb92ecb554287977b7a980bd4d2b4
I20240501 08:13:39.317409 26028 log_verifier.cc:126] Checking tablet a9d4354546c040d6b971ed6fc3b02c76
I20240501 08:13:39.665372 26028 log_verifier.cc:177] Verified matching terms for 1073 ops in tablet a9d4354546c040d6b971ed6fc3b02c76
I20240501 08:13:39.666754 26028 log_verifier.cc:126] Checking tablet ad5a9e14873742caa50ec838cca1c715
I20240501 08:13:40.015569 26028 log_verifier.cc:177] Verified matching terms for 1161 ops in tablet ad5a9e14873742caa50ec838cca1c715
I20240501 08:13:40.016942 26028 log_verifier.cc:126] Checking tablet adbf537bdcdf4443a94799340aab4205
I20240501 08:13:40.367273 26028 log_verifier.cc:177] Verified matching terms for 1101 ops in tablet adbf537bdcdf4443a94799340aab4205
I20240501 08:13:40.368546 26028 log_verifier.cc:126] Checking tablet b4c8611ac7394bbd96cabee647f5c724
I20240501 08:13:40.709033 26028 log_verifier.cc:177] Verified matching terms for 1139 ops in tablet b4c8611ac7394bbd96cabee647f5c724
I20240501 08:13:40.710331 26028 log_verifier.cc:126] Checking tablet db3c2a3262974c75bba7743192d66d46
I20240501 08:13:41.085479 26028 log_verifier.cc:177] Verified matching terms for 1139 ops in tablet db3c2a3262974c75bba7743192d66d46
I20240501 08:13:41.086815 26028 log_verifier.cc:126] Checking tablet ddaae299e96b412282bec452f9b06fa2
I20240501 08:13:41.433490 26028 log_verifier.cc:177] Verified matching terms for 1193 ops in tablet ddaae299e96b412282bec452f9b06fa2
I20240501 08:13:41.434826 26028 log_verifier.cc:126] Checking tablet f4ec90b4ddce408a81f588fa7546e0dd
I20240501 08:13:41.820884 26028 log_verifier.cc:177] Verified matching terms for 1172 ops in tablet f4ec90b4ddce408a81f588fa7546e0dd
I20240501 08:13:41.825362 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 32246
I20240501 08:13:41.844595 32368 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:42.179654 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 32246
W20240501 08:13:42.240639 1359 connection.cc:535] server connection from 127.25.107.1:55683 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:42.240907 1227 connection.cc:535] server connection from 127.25.107.1:45063 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:42.241303 32393 connection.cc:535] server connection from 127.25.107.1:48169 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:13:42.241624 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 32377
I20240501 08:13:42.247380 32498 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:42.545732 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 32377
W20240501 08:13:42.598753 1095 connection.cc:535] server connection from 127.25.107.2:58795 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:42.598821 32171 connection.cc:535] server connection from 127.25.107.2:49371 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:13:42.599068 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 32507
I20240501 08:13:42.604481 32629 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:13:42.721086 1488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.2:46875: connect: Connection refused (error 111)
W20240501 08:13:42.722502 1488 consensus_peers.cc:485] T a9d4354546c040d6b971ed6fc3b02c76 P c5e88f3df73d4d4f996304b6df7e6448 -> Peer b3e6c8e1eb974b5c9c8b7e109b214576 (127.25.107.2:46875): Couldn't send request to peer b3e6c8e1eb974b5c9c8b7e109b214576. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.2:46875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:42.909296 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 32507
I20240501 08:13:42.961761 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 1080
I20240501 08:13:42.967846 1201 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:43.236490 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 1080
I20240501 08:13:43.285344 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 1211
I20240501 08:13:43.291082 1332 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:13:43.431658 5166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.4:39729: connect: Connection refused (error 111)
W20240501 08:13:43.435531 5166 consensus_peers.cc:485] T b4c8611ac7394bbd96cabee647f5c724 P 5341ea9c08294eb1af0f2fd36c1bec7b -> Peer 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729): Couldn't send request to peer 489550c4b21146918c1c9618be65b4ff. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.4:39729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:43.557307 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 1211
W20240501 08:13:43.563671 1489 consensus_peers.cc:485] T a50cb92ecb554287977b7a980bd4d2b4 P c5e88f3df73d4d4f996304b6df7e6448 -> Peer 489550c4b21146918c1c9618be65b4ff (127.25.107.4:39729): Couldn't send request to peer 489550c4b21146918c1c9618be65b4ff. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.4:39729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:43.605978 1490 connection.cc:535] server connection from 127.25.107.5:43981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:13:43.606385 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 1341
I20240501 08:13:43.612144 1462 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:43.871892 6453 raft_consensus.cc:482] T 841db634d09c4ef0bdfda733a489af5d P 37965efda2a34916b49a2357eab43bf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2219c1853e76474799f93fed12cdc12d)
I20240501 08:13:43.872121 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 1341
I20240501 08:13:43.920496 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 1471
I20240501 08:13:43.925843 1598 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:13:44.019665 5165 consensus_peers.cc:485] T 5dbd5afc230c41509774812f082f97f0 P 5341ea9c08294eb1af0f2fd36c1bec7b -> Peer 37965efda2a34916b49a2357eab43bf3 (127.25.107.6:44049): Couldn't send request to peer 37965efda2a34916b49a2357eab43bf3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:44.191275 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 1471
W20240501 08:13:44.237380 32174 connection.cc:535] server connection from 127.25.107.7:54789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:13:44.237705 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 5146
I20240501 08:13:44.243646 5271 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:44.438738 5851 raft_consensus.cc:482] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e44d7017614f88b3205111d85de021)
I20240501 08:13:44.438957 5851 raft_consensus.cc:504] T 50c58a563cf94527895a675a1e729fe8 P 5341ea9c08294eb1af0f2fd36c1bec7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 1196 OBSOLETE_local: false peers { permanent_uuid: "12e44d7017614f88b3205111d85de021" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 38795 } } peers { permanent_uuid: "5341ea9c08294eb1af0f2fd36c1bec7b" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 46277 } }
I20240501 08:13:44.439242 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 5146
I20240501 08:13:44.469928 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 32156
I20240501 08:13:44.470978 32217 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:44.716017 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 32156
2024-05-01T08:13:44Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (115184 ms)
[----------] 1 test from RebalanceParamTest (115184 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240501 08:13:44.829663 26028 test_util.cc:268] Using random seed: 280266437
I20240501 08:13:44.831627 26028 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:13:44.831705 26028 ts_itest-base.cc:113] --------------
I20240501 08:13:44.831753 26028 ts_itest-base.cc:114] 8 tablet servers
I20240501 08:13:44.831802 26028 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:13:44.831856 26028 ts_itest-base.cc:116] --------------
2024-05-01T08:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:13:44Z Disabled control of system clock
I20240501 08:13:44.904763 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.107.62:36223
--webserver_interface=127.25.107.62
--webserver_port=0
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.107.62:36223
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 08:13:45.273356 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:45.273694 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:45.273833 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:45.281710 6466 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:13:45.281864 6466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:13:45.281926 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:45.282007 6466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:13:45.282080 6466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:13:45.292310 6466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.25.107.62:36223
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.107.62:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.107.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:45.293715 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:45.295640 6466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:45.306811 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:45.306779 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:45.306864 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:45.307786 6466 server_base.cc:1034] running on GCE node
I20240501 08:13:45.308662 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:45.310390 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:45.311658 6466 hybrid_clock.cc:648] HybridClock initialized: now 1714551225311589 us; error 55 us; skew 500 ppm
I20240501 08:13:45.312065 6466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:45.315860 6466 webserver.cc:460] Webserver started at http://127.25.107.62:34953/ using document root <none> and password file <none>
I20240501 08:13:45.316730 6466 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:45.316879 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:45.317239 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:45.320204 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d978dc7272e947d9938cb39d0d6c1eb8"
format_stamp: "Formatted at 2024-05-01 08:13:45 on dist-test-slave-dv41"
I20240501 08:13:45.321106 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d978dc7272e947d9938cb39d0d6c1eb8"
format_stamp: "Formatted at 2024-05-01 08:13:45 on dist-test-slave-dv41"
I20240501 08:13:45.327723 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240501 08:13:45.331952 6480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:45.333693 6466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240501 08:13:45.333928 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d978dc7272e947d9938cb39d0d6c1eb8"
format_stamp: "Formatted at 2024-05-01 08:13:45 on dist-test-slave-dv41"
I20240501 08:13:45.334200 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:45.369050 6466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:45.369993 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:45.370337 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:45.388818 6466 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.62:36223
I20240501 08:13:45.388844 6531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.62:36223 every 8 connection(s)
I20240501 08:13:45.390312 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:13:45.394562 6532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:45.397836 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 6466
I20240501 08:13:45.398110 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:13:45.407446 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8: Bootstrap starting.
I20240501 08:13:45.411340 6532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:45.412652 6532 log.cc:824] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:45.415692 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8: No bootstrap required, opened a new log
I20240501 08:13:45.422274 6532 raft_consensus.cc:348] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d978dc7272e947d9938cb39d0d6c1eb8" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 36223 } }
I20240501 08:13:45.422708 6532 raft_consensus.cc:374] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:45.422816 6532 raft_consensus.cc:729] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d978dc7272e947d9938cb39d0d6c1eb8, State: Initialized, Role: FOLLOWER
I20240501 08:13:45.423502 6532 consensus_queue.cc:260] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d978dc7272e947d9938cb39d0d6c1eb8" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 36223 } }
I20240501 08:13:45.423709 6532 raft_consensus.cc:388] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:13:45.423842 6532 raft_consensus.cc:482] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:13:45.424001 6532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:45.426241 6532 raft_consensus.cc:504] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d978dc7272e947d9938cb39d0d6c1eb8" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 36223 } }
I20240501 08:13:45.426865 6532 leader_election.cc:302] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d978dc7272e947d9938cb39d0d6c1eb8; no voters:
I20240501 08:13:45.427474 6532 leader_election.cc:288] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:13:45.427636 6537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:45.428254 6537 raft_consensus.cc:686] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [term 1 LEADER]: Becoming Leader. State: Replica: d978dc7272e947d9938cb39d0d6c1eb8, State: Running, Role: LEADER
I20240501 08:13:45.428830 6537 consensus_queue.cc:237] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d978dc7272e947d9938cb39d0d6c1eb8" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 36223 } }
I20240501 08:13:45.429561 6532 sys_catalog.cc:564] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:13:45.431684 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d978dc7272e947d9938cb39d0d6c1eb8. Latest consensus state: current_term: 1 leader_uuid: "d978dc7272e947d9938cb39d0d6c1eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d978dc7272e947d9938cb39d0d6c1eb8" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 36223 } } }
I20240501 08:13:45.432154 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [sys.catalog]: This master's current role is: LEADER
I20240501 08:13:45.433076 6538 sys_catalog.cc:455] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d978dc7272e947d9938cb39d0d6c1eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d978dc7272e947d9938cb39d0d6c1eb8" member_type: VOTER last_known_addr { host: "127.25.107.62" port: 36223 } } }
I20240501 08:13:45.433362 6538 sys_catalog.cc:458] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8 [sys.catalog]: This master's current role is: LEADER
I20240501 08:13:45.434576 6542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:13:45.440871 6542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:13:45.451018 6542 catalog_manager.cc:1347] Generated new cluster ID: d0bda095aeef4d3ebfc177151fb9e4f0
I20240501 08:13:45.451154 6542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:13:45.466492 6542 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:13:45.467778 6542 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:13:45.482949 6542 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d978dc7272e947d9938cb39d0d6c1eb8: Generated new TSK 0
I20240501 08:13:45.483935 6542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:13:45.494972 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.1:0
--local_ip_for_outbound_sockets=127.25.107.1
--webserver_interface=127.25.107.1
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:13:45.867115 6556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:45.867431 6556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:45.867522 6556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:45.875363 6556 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:13:45.875502 6556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:45.875628 6556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.1
I20240501 08:13:45.885041 6556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.107.1
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.1
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:45.886441 6556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:45.888243 6556 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:45.900908 6561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:45.900956 6564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:45.900990 6562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:45.901770 6556 server_base.cc:1034] running on GCE node
I20240501 08:13:45.902484 6556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:45.903544 6556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:45.904778 6556 hybrid_clock.cc:648] HybridClock initialized: now 1714551225904722 us; error 53 us; skew 500 ppm
I20240501 08:13:45.905100 6556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:45.907663 6556 webserver.cc:460] Webserver started at http://127.25.107.1:45931/ using document root <none> and password file <none>
I20240501 08:13:45.908476 6556 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:45.908588 6556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:45.908896 6556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:45.911767 6556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "97cdee0bf9d84645a1451f9822c5d699"
format_stamp: "Formatted at 2024-05-01 08:13:45 on dist-test-slave-dv41"
I20240501 08:13:45.912715 6556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "97cdee0bf9d84645a1451f9822c5d699"
format_stamp: "Formatted at 2024-05-01 08:13:45 on dist-test-slave-dv41"
I20240501 08:13:45.918841 6556 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240501 08:13:45.922806 6570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:45.924407 6556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:13:45.924603 6556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "97cdee0bf9d84645a1451f9822c5d699"
format_stamp: "Formatted at 2024-05-01 08:13:45 on dist-test-slave-dv41"
I20240501 08:13:45.924809 6556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:45.967676 6556 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:45.968590 6556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:45.968850 6556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:45.969862 6556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:45.973629 6556 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:45.974061 6556 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.002s sys 0.000s
I20240501 08:13:45.974406 6556 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:45.974691 6556 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:46.012341 6556 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.1:45193
I20240501 08:13:46.012401 6682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.1:45193 every 8 connection(s)
I20240501 08:13:46.013818 6556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:13:46.017666 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 6556
I20240501 08:13:46.017917 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:13:46.021397 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.2:0
--local_ip_for_outbound_sockets=127.25.107.2
--webserver_interface=127.25.107.2
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:13:46.028164 6683 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:13:46.028520 6683 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:46.029475 6683 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:13:46.031981 6497 ts_manager.cc:194] Registered new tserver with Master: 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:46.033881 6497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.1:46911
W20240501 08:13:46.389022 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:46.389333 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:46.389446 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:46.397387 6687 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:13:46.397536 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:46.397687 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.2
I20240501 08:13:46.407510 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.107.2
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.2
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:46.408973 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:46.410935 6687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:46.423727 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:46.423774 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:46.423800 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:46.424993 6687 server_base.cc:1034] running on GCE node
I20240501 08:13:46.425643 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:46.426724 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:46.428007 6687 hybrid_clock.cc:648] HybridClock initialized: now 1714551226427947 us; error 54 us; skew 500 ppm
I20240501 08:13:46.428372 6687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:46.431082 6687 webserver.cc:460] Webserver started at http://127.25.107.2:38883/ using document root <none> and password file <none>
I20240501 08:13:46.431875 6687 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:46.432003 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:46.432325 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:46.435274 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cd3434d87b3c46c6848a711cdf6b150c"
format_stamp: "Formatted at 2024-05-01 08:13:46 on dist-test-slave-dv41"
I20240501 08:13:46.436177 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cd3434d87b3c46c6848a711cdf6b150c"
format_stamp: "Formatted at 2024-05-01 08:13:46 on dist-test-slave-dv41"
I20240501 08:13:46.442304 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240501 08:13:46.445971 6701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:46.447446 6687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:46.447638 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cd3434d87b3c46c6848a711cdf6b150c"
format_stamp: "Formatted at 2024-05-01 08:13:46 on dist-test-slave-dv41"
I20240501 08:13:46.447863 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:46.469571 6687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:46.470600 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:46.470898 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:46.471932 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:46.473604 6687 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:46.473718 6687 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:46.473847 6687 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:46.473922 6687 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:46.505867 6687 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.2:46507
I20240501 08:13:46.505913 6813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.2:46507 every 8 connection(s)
I20240501 08:13:46.507380 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:13:46.515652 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 6687
I20240501 08:13:46.515925 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:13:46.519722 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.3:0
--local_ip_for_outbound_sockets=127.25.107.3
--webserver_interface=127.25.107.3
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:13:46.522114 6814 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:13:46.522486 6814 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:46.523308 6814 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:13:46.524864 6497 ts_manager.cc:194] Registered new tserver with Master: cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:13:46.525853 6497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.2:56173
W20240501 08:13:46.889918 6817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:46.890286 6817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:46.890420 6817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:46.898180 6817 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:13:46.898360 6817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:46.898543 6817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.3
I20240501 08:13:46.908121 6817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.107.3
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.3
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:46.909510 6817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:46.911438 6817 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:46.924042 6825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:46.923983 6822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:46.923946 6823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:46.924886 6817 server_base.cc:1034] running on GCE node
I20240501 08:13:46.925565 6817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:46.926761 6817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:46.927986 6817 hybrid_clock.cc:648] HybridClock initialized: now 1714551226927915 us; error 59 us; skew 500 ppm
I20240501 08:13:46.928326 6817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:46.936605 6817 webserver.cc:460] Webserver started at http://127.25.107.3:37157/ using document root <none> and password file <none>
I20240501 08:13:46.937474 6817 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:46.937605 6817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:46.937983 6817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:46.943275 6817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
format_stamp: "Formatted at 2024-05-01 08:13:46 on dist-test-slave-dv41"
I20240501 08:13:46.944150 6817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
format_stamp: "Formatted at 2024-05-01 08:13:46 on dist-test-slave-dv41"
I20240501 08:13:46.950235 6817 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20240501 08:13:46.954447 6831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:46.955988 6817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:13:46.956197 6817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
format_stamp: "Formatted at 2024-05-01 08:13:46 on dist-test-slave-dv41"
I20240501 08:13:46.956432 6817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:46.975920 6817 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:46.976886 6817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:46.977180 6817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:46.978276 6817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:46.979959 6817 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:46.980067 6817 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:46.980180 6817 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:46.980248 6817 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:47.012506 6817 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.3:43505
I20240501 08:13:47.012550 6943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.3:43505 every 8 connection(s)
I20240501 08:13:47.013996 6817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:13:47.014483 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 6817
I20240501 08:13:47.014693 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:13:47.018733 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.4:0
--local_ip_for_outbound_sockets=127.25.107.4
--webserver_interface=127.25.107.4
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:13:47.036876 6944 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:13:47.037321 6944 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:47.037360 6683 heartbeater.cc:503] Master 127.25.107.62:36223 was elected leader, sending a full tablet report...
I20240501 08:13:47.038435 6944 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:13:47.040160 6497 ts_manager.cc:194] Registered new tserver with Master: e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:47.041002 6497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.3:57727
W20240501 08:13:47.406406 6947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:47.406759 6947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:47.406880 6947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:47.414788 6947 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:13:47.414943 6947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:47.415127 6947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.4
I20240501 08:13:47.424916 6947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.107.4
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.4
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:47.426466 6947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:47.428571 6947 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:47.443298 6955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:47.443352 6952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:47.443367 6953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:47.444772 6947 server_base.cc:1034] running on GCE node
I20240501 08:13:47.445542 6947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:47.446779 6947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:47.448102 6947 hybrid_clock.cc:648] HybridClock initialized: now 1714551227448022 us; error 58 us; skew 500 ppm
I20240501 08:13:47.448732 6947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:47.452095 6947 webserver.cc:460] Webserver started at http://127.25.107.4:37537/ using document root <none> and password file <none>
I20240501 08:13:47.453058 6947 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:47.453197 6947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:47.453603 6947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:47.456631 6947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "28b7613bd4834f3ab245c82dd7e80240"
format_stamp: "Formatted at 2024-05-01 08:13:47 on dist-test-slave-dv41"
I20240501 08:13:47.457576 6947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "28b7613bd4834f3ab245c82dd7e80240"
format_stamp: "Formatted at 2024-05-01 08:13:47 on dist-test-slave-dv41"
I20240501 08:13:47.465094 6947 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20240501 08:13:47.469502 6961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:47.471341 6947 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 08:13:47.471555 6947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "28b7613bd4834f3ab245c82dd7e80240"
format_stamp: "Formatted at 2024-05-01 08:13:47 on dist-test-slave-dv41"
I20240501 08:13:47.471788 6947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:47.499712 6947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:47.500777 6947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:47.501127 6947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:47.502449 6947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:47.504364 6947 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:47.504478 6947 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:47.504596 6947 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:47.504667 6947 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:47.529371 6814 heartbeater.cc:503] Master 127.25.107.62:36223 was elected leader, sending a full tablet report...
I20240501 08:13:47.554659 6947 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.4:43623
I20240501 08:13:47.554734 7073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.4:43623 every 8 connection(s)
I20240501 08:13:47.556521 6947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:13:47.559262 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 6947
I20240501 08:13:47.559548 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:13:47.564757 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.5:0
--local_ip_for_outbound_sockets=127.25.107.5
--webserver_interface=127.25.107.5
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:13:47.580382 7074 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:13:47.580829 7074 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:47.581923 7074 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:13:47.583940 6497 ts_manager.cc:194] Registered new tserver with Master: 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:13:47.585160 6497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.4:59553
W20240501 08:13:47.955199 7078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:47.955524 7078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:47.955614 7078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:47.963675 7078 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:13:47.963833 7078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:47.963968 7078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.5
I20240501 08:13:47.973968 7078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.107.5
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.5
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:47.975468 7078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:47.977892 7078 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:47.993669 7086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:47.993786 7084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:47.993759 7083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:47.994410 7078 server_base.cc:1034] running on GCE node
I20240501 08:13:47.995093 7078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:47.996181 7078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:47.997421 7078 hybrid_clock.cc:648] HybridClock initialized: now 1714551227997356 us; error 66 us; skew 500 ppm
I20240501 08:13:47.997839 7078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:48.000562 7078 webserver.cc:460] Webserver started at http://127.25.107.5:37091/ using document root <none> and password file <none>
I20240501 08:13:48.001422 7078 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:48.001556 7078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:48.001945 7078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:48.004956 7078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0a814bba1abe4653bcd418a83ec42370"
format_stamp: "Formatted at 2024-05-01 08:13:47 on dist-test-slave-dv41"
I20240501 08:13:48.005887 7078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0a814bba1abe4653bcd418a83ec42370"
format_stamp: "Formatted at 2024-05-01 08:13:47 on dist-test-slave-dv41"
I20240501 08:13:48.012218 7078 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240501 08:13:48.016227 7092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:48.017798 7078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 08:13:48.017990 7078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0a814bba1abe4653bcd418a83ec42370"
format_stamp: "Formatted at 2024-05-01 08:13:47 on dist-test-slave-dv41"
I20240501 08:13:48.018198 7078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:48.033339 7078 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:48.034353 7078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:48.034621 7078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:48.035682 7078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:48.037395 7078 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:48.037487 7078 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:48.037575 7078 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:48.037627 7078 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:48.043957 6944 heartbeater.cc:503] Master 127.25.107.62:36223 was elected leader, sending a full tablet report...
I20240501 08:13:48.072948 7078 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.5:45691
I20240501 08:13:48.073220 7204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.5:45691 every 8 connection(s)
I20240501 08:13:48.074579 7078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:13:48.083820 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 7078
I20240501 08:13:48.084014 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:13:48.088066 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.6:0
--local_ip_for_outbound_sockets=127.25.107.6
--webserver_interface=127.25.107.6
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:13:48.089972 7205 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:13:48.090335 7205 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:48.091190 7205 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:13:48.092903 6497 ts_manager.cc:194] Registered new tserver with Master: 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:13:48.093825 6497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.5:44405
W20240501 08:13:48.482852 7208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:48.483188 7208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:48.483278 7208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:48.491032 7208 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:13:48.491179 7208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:48.491307 7208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.6
I20240501 08:13:48.500896 7208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.107.6
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.6
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:48.502482 7208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:48.504328 7208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:48.517388 7214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:48.517513 7213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:48.517987 7216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:48.518443 7208 server_base.cc:1034] running on GCE node
I20240501 08:13:48.519117 7208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:48.520229 7208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:48.521457 7208 hybrid_clock.cc:648] HybridClock initialized: now 1714551228521388 us; error 67 us; skew 500 ppm
I20240501 08:13:48.521826 7208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:48.524678 7208 webserver.cc:460] Webserver started at http://127.25.107.6:33059/ using document root <none> and password file <none>
I20240501 08:13:48.525553 7208 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:48.525672 7208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:48.526013 7208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:48.529196 7208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "aba7c226aaca4e8c90931f547d48aaa3"
format_stamp: "Formatted at 2024-05-01 08:13:48 on dist-test-slave-dv41"
I20240501 08:13:48.530216 7208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "aba7c226aaca4e8c90931f547d48aaa3"
format_stamp: "Formatted at 2024-05-01 08:13:48 on dist-test-slave-dv41"
I20240501 08:13:48.536511 7208 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240501 08:13:48.540648 7222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:48.542230 7208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:48.542459 7208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "aba7c226aaca4e8c90931f547d48aaa3"
format_stamp: "Formatted at 2024-05-01 08:13:48 on dist-test-slave-dv41"
I20240501 08:13:48.542686 7208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:48.568470 7208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:48.569443 7208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:48.569743 7208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:48.570876 7208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:48.572552 7208 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:48.572654 7208 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:48.572770 7208 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:48.572837 7208 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:48.588526 7074 heartbeater.cc:503] Master 127.25.107.62:36223 was elected leader, sending a full tablet report...
I20240501 08:13:48.605170 7208 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.6:41219
I20240501 08:13:48.605221 7334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.6:41219 every 8 connection(s)
I20240501 08:13:48.606653 7208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:13:48.610045 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 7208
I20240501 08:13:48.610309 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:13:48.613744 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.7:0
--local_ip_for_outbound_sockets=127.25.107.7
--webserver_interface=127.25.107.7
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:13:48.620952 7335 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:13:48.621311 7335 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:48.622123 7335 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:13:48.623834 6497 ts_manager.cc:194] Registered new tserver with Master: aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
I20240501 08:13:48.624814 6497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.6:41407
W20240501 08:13:48.980268 7339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:48.980655 7339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:48.980937 7339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:48.989030 7339 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:13:48.989212 7339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:48.989377 7339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.7
I20240501 08:13:48.999130 7339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.107.7
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.7
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:49.000614 7339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:49.002547 7339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:49.015689 7345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:49.015897 7344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:49.015779 7347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:49.016714 7339 server_base.cc:1034] running on GCE node
I20240501 08:13:49.017567 7339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:49.018810 7339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:49.020040 7339 hybrid_clock.cc:648] HybridClock initialized: now 1714551229019975 us; error 58 us; skew 500 ppm
I20240501 08:13:49.020378 7339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:49.022890 7339 webserver.cc:460] Webserver started at http://127.25.107.7:35309/ using document root <none> and password file <none>
I20240501 08:13:49.023741 7339 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:49.023857 7339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:49.024147 7339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:49.027062 7339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a518ca5e9de24204bd028b0ce4a77d58"
format_stamp: "Formatted at 2024-05-01 08:13:49 on dist-test-slave-dv41"
I20240501 08:13:49.027913 7339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a518ca5e9de24204bd028b0ce4a77d58"
format_stamp: "Formatted at 2024-05-01 08:13:49 on dist-test-slave-dv41"
I20240501 08:13:49.034051 7339 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240501 08:13:49.037928 7353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:49.039573 7339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:49.039796 7339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a518ca5e9de24204bd028b0ce4a77d58"
format_stamp: "Formatted at 2024-05-01 08:13:49 on dist-test-slave-dv41"
I20240501 08:13:49.040020 7339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:49.074705 7339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:49.075618 7339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:49.075918 7339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:49.077127 7339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:49.078856 7339 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:49.078971 7339 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:49.079088 7339 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:49.079157 7339 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:49.096647 7205 heartbeater.cc:503] Master 127.25.107.62:36223 was elected leader, sending a full tablet report...
I20240501 08:13:49.110543 7339 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.7:33929
I20240501 08:13:49.110607 7465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.7:33929 every 8 connection(s)
I20240501 08:13:49.112022 7339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:13:49.117930 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 7339
I20240501 08:13:49.118114 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:13:49.121906 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.8:0
--local_ip_for_outbound_sockets=127.25.107.8
--webserver_interface=127.25.107.8
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:13:49.126842 7466 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:13:49.127260 7466 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:49.128123 7466 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:13:49.129863 6497 ts_manager.cc:194] Registered new tserver with Master: a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929)
I20240501 08:13:49.131093 6497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.7:57105
W20240501 08:13:49.495896 7469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:49.496243 7469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:49.496345 7469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:49.504072 7469 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:13:49.504215 7469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:49.504379 7469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.8
I20240501 08:13:49.514605 7469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.25.107.8
--webserver_port=0
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.8
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:49.516032 7469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:49.517944 7469 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:49.531091 7475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:49.531132 7474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:49.531167 7477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:49.531940 7469 server_base.cc:1034] running on GCE node
I20240501 08:13:49.532691 7469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:49.533716 7469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:49.534929 7469 hybrid_clock.cc:648] HybridClock initialized: now 1714551229534874 us; error 56 us; skew 500 ppm
I20240501 08:13:49.535312 7469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:49.538172 7469 webserver.cc:460] Webserver started at http://127.25.107.8:37217/ using document root <none> and password file <none>
I20240501 08:13:49.539121 7469 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:49.539250 7469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:49.539592 7469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:49.542443 7469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "19fdb76c8d1f41d7842e397adfc1ef3f"
format_stamp: "Formatted at 2024-05-01 08:13:49 on dist-test-slave-dv41"
I20240501 08:13:49.543334 7469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "19fdb76c8d1f41d7842e397adfc1ef3f"
format_stamp: "Formatted at 2024-05-01 08:13:49 on dist-test-slave-dv41"
I20240501 08:13:49.549571 7469 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240501 08:13:49.554013 7483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:49.555722 7469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20240501 08:13:49.555933 7469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "19fdb76c8d1f41d7842e397adfc1ef3f"
format_stamp: "Formatted at 2024-05-01 08:13:49 on dist-test-slave-dv41"
I20240501 08:13:49.556164 7469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:49.587705 7469 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:49.588693 7469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:49.589013 7469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:49.590070 7469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:49.592234 7469 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:49.592367 7469 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:49.592496 7469 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:49.592581 7469 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:49.624378 7469 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.8:41625
I20240501 08:13:49.624435 7595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.8:41625 every 8 connection(s)
I20240501 08:13:49.625963 7469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:13:49.627604 7335 heartbeater.cc:503] Master 127.25.107.62:36223 was elected leader, sending a full tablet report...
I20240501 08:13:49.629112 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 7469
I20240501 08:13:49.629323 26028 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:13:49.640394 7596 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:13:49.640748 7596 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:49.641616 7596 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:13:49.643201 6497 ts_manager.cc:194] Registered new tserver with Master: 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625)
I20240501 08:13:49.644064 6497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.8:44085
I20240501 08:13:49.646222 26028 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:13:49.664533 26028 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:13:49.664700 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 7078
I20240501 08:13:49.673825 7200 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:49.756778 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 7078
I20240501 08:13:49.772574 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 7208
I20240501 08:13:49.781168 7330 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:49.862768 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 7208
I20240501 08:13:49.877454 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 7339
I20240501 08:13:49.885810 7461 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:49.967170 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 7339
I20240501 08:13:49.984449 26028 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu pid 7469
I20240501 08:13:49.995033 7591 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:50.079676 26028 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu with pid 7469
I20240501 08:13:53.846421 6496 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42016:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:53.891316 6749 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7f87b6032d45c5be5d748ab2cc71e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:53.891590 6748 tablet_service.cc:1467] Processing CreateTablet for tablet 070e927b5d684367ae0b40b12a23da54 (DEFAULT_TABLE table=rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:53.893465 6749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7f87b6032d45c5be5d748ab2cc71e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:53.894035 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070e927b5d684367ae0b40b12a23da54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:53.896168 6618 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7f87b6032d45c5be5d748ab2cc71e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:53.897814 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7f87b6032d45c5be5d748ab2cc71e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:53.898416 6878 tablet_service.cc:1467] Processing CreateTablet for tablet 5f090fda814f455a88dccd6680bdd05f (DEFAULT_TABLE table=rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:53.899909 6878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f090fda814f455a88dccd6680bdd05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:53.902810 6617 tablet_service.cc:1467] Processing CreateTablet for tablet 070e927b5d684367ae0b40b12a23da54 (DEFAULT_TABLE table=rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:53.903388 6617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070e927b5d684367ae0b40b12a23da54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:53.904543 6879 tablet_service.cc:1467] Processing CreateTablet for tablet 070e927b5d684367ae0b40b12a23da54 (DEFAULT_TABLE table=rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:53.905158 6879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070e927b5d684367ae0b40b12a23da54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:53.907070 7009 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7f87b6032d45c5be5d748ab2cc71e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:53.894937 6616 tablet_service.cc:1467] Processing CreateTablet for tablet 5f090fda814f455a88dccd6680bdd05f (DEFAULT_TABLE table=rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:53.908546 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f090fda814f455a88dccd6680bdd05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:53.908682 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7f87b6032d45c5be5d748ab2cc71e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:53.910817 7008 tablet_service.cc:1467] Processing CreateTablet for tablet 5f090fda814f455a88dccd6680bdd05f (DEFAULT_TABLE table=rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:53.911635 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f090fda814f455a88dccd6680bdd05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:53.943322 7628 tablet_bootstrap.cc:492] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:53.944029 7629 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:53.947072 7626 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:53.947309 7627 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:53.948832 7628 tablet_bootstrap.cc:654] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:53.950105 7629 tablet_bootstrap.cc:654] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:53.950306 7628 log.cc:824] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:53.950888 7627 tablet_bootstrap.cc:654] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:53.950888 7626 tablet_bootstrap.cc:654] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:53.951572 7629 log.cc:824] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:53.952216 7626 log.cc:824] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:53.952275 7627 log.cc:824] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:53.960327 7628 tablet_bootstrap.cc:492] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:53.960711 7628 ts_tablet_manager.cc:1367] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.006s
I20240501 08:13:53.960840 7629 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:53.961216 7629 ts_tablet_manager.cc:1367] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20240501 08:13:53.969327 7626 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:53.969331 7627 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:53.969699 7626 ts_tablet_manager.cc:1367] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.015s
I20240501 08:13:53.969703 7627 ts_tablet_manager.cc:1367] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.007s
I20240501 08:13:53.970470 7629 raft_consensus.cc:348] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:53.971004 7629 raft_consensus.cc:374] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:53.971124 7629 raft_consensus.cc:729] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:53.971205 7628 raft_consensus.cc:348] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:53.971809 7628 raft_consensus.cc:374] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:53.971954 7628 raft_consensus.cc:729] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:53.972075 7629 consensus_queue.cc:260] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:53.972841 7628 consensus_queue.cc:260] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:53.981233 7629 ts_tablet_manager.cc:1398] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.020s user 0.012s sys 0.005s
I20240501 08:13:53.981387 7627 raft_consensus.cc:348] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:53.981947 7627 raft_consensus.cc:374] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:53.982077 7627 raft_consensus.cc:729] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:53.982692 7626 raft_consensus.cc:348] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:53.983633 7626 raft_consensus.cc:374] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:53.983814 7626 raft_consensus.cc:729] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:53.985411 7626 consensus_queue.cc:260] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:53.990311 7627 consensus_queue.cc:260] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:53.990583 7628 ts_tablet_manager.cc:1398] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.030s user 0.026s sys 0.003s
I20240501 08:13:53.991312 7628 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:53.989254 7629 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:53.994992 7629 tablet_bootstrap.cc:654] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:53.997871 7629 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:53.998056 7629 ts_tablet_manager.cc:1367] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 08:13:53.999171 7629 raft_consensus.cc:348] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:53.999526 7629 raft_consensus.cc:374] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:53.999622 7629 raft_consensus.cc:729] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:53.999758 7627 ts_tablet_manager.cc:1398] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.030s user 0.009s sys 0.006s
I20240501 08:13:54.000291 7627 tablet_bootstrap.cc:492] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:54.002071 7627 tablet_bootstrap.cc:654] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:54.002120 7628 tablet_bootstrap.cc:654] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:53.999938 7629 consensus_queue.cc:260] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.004801 7629 ts_tablet_manager.cc:1398] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240501 08:13:54.008474 7626 ts_tablet_manager.cc:1398] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.039s user 0.019s sys 0.008s
I20240501 08:13:54.008993 7626 tablet_bootstrap.cc:492] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:54.010694 7627 tablet_bootstrap.cc:492] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:54.010866 7627 ts_tablet_manager.cc:1367] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 08:13:54.011955 7627 raft_consensus.cc:348] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.012187 7627 raft_consensus.cc:374] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:54.012276 7627 raft_consensus.cc:729] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:54.013392 7626 tablet_bootstrap.cc:654] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
W20240501 08:13:54.015091 6815 tablet.cc:2369] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:13:54.015838 7627 consensus_queue.cc:260] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.016570 7627 ts_tablet_manager.cc:1398] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20240501 08:13:54.022784 6684 tablet.cc:2369] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:13:54.024152 7628 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:54.024338 7628 ts_tablet_manager.cc:1367] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.007s
I20240501 08:13:54.025260 7626 tablet_bootstrap.cc:492] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:54.025245 7628 raft_consensus.cc:348] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.025435 7626 ts_tablet_manager.cc:1367] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20240501 08:13:54.025444 7628 raft_consensus.cc:374] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:54.025534 7628 raft_consensus.cc:729] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:54.025782 7628 consensus_queue.cc:260] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
W20240501 08:13:54.026438 6945 tablet.cc:2369] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:13:54.026459 7626 raft_consensus.cc:348] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.026660 7626 raft_consensus.cc:374] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:54.026765 7626 raft_consensus.cc:729] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:54.027163 7628 ts_tablet_manager.cc:1398] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:13:54.027465 7628 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:54.027016 7626 consensus_queue.cc:260] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.028846 7626 ts_tablet_manager.cc:1398] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:54.029737 7628 tablet_bootstrap.cc:654] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:54.035506 7628 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:54.035710 7628 ts_tablet_manager.cc:1367] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 08:13:54.036346 7628 raft_consensus.cc:348] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:54.036549 7628 raft_consensus.cc:374] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:54.036623 7628 raft_consensus.cc:729] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:54.036852 7628 consensus_queue.cc:260] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:54.037323 7628 ts_tablet_manager.cc:1398] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240501 08:13:54.064873 7075 tablet.cc:2369] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:13:54.096419 7634 raft_consensus.cc:482] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:54.096691 7634 raft_consensus.cc:504] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:54.098603 7634 leader_election.cc:288] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:13:54.107955 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7f87b6032d45c5be5d748ab2cc71e2" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:13:54.107955 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7f87b6032d45c5be5d748ab2cc71e2" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:13:54.108588 6769 raft_consensus.cc:2454] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:13:54.108592 6638 raft_consensus.cc:2454] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:13:54.109381 6962 leader_election.cc:302] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:13:54.109854 7634 raft_consensus.cc:2781] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:54.110006 7634 raft_consensus.cc:482] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:54.110098 7634 raft_consensus.cc:3037] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:54.112283 7634 raft_consensus.cc:504] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:54.112936 7634 leader_election.cc:288] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Requested vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:13:54.113493 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7f87b6032d45c5be5d748ab2cc71e2" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:13:54.113523 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7f87b6032d45c5be5d748ab2cc71e2" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:13:54.113687 6638 raft_consensus.cc:3037] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:54.113710 6769 raft_consensus.cc:3037] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:54.116209 6638 raft_consensus.cc:2454] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:13:54.116228 6769 raft_consensus.cc:2454] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:13:54.116710 6965 leader_election.cc:302] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:13:54.117043 7634 raft_consensus.cc:2781] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:54.117460 7634 raft_consensus.cc:686] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:13:54.117982 7634 consensus_queue.cc:237] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:54.121362 7634 raft_consensus.cc:482] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:54.121635 7634 raft_consensus.cc:504] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.122362 7634 leader_election.cc:288] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:54.122694 6496 catalog_manager.cc:5484] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:13:54.123219 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:13:54.123483 6638 raft_consensus.cc:2454] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:13:54.123934 6962 leader_election.cc:302] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:13:54.125661 7634 raft_consensus.cc:2781] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:54.125782 7634 raft_consensus.cc:482] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:54.125895 7634 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:54.128579 7634 raft_consensus.cc:504] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.129326 7634 leader_election.cc:288] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Requested vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:54.129786 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:13:54.129810 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:13:54.130395 6899 raft_consensus.cc:2454] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:13:54.130487 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:13:54.130733 6638 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:54.133216 6638 raft_consensus.cc:2454] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:13:54.133694 6962 leader_election.cc:302] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters: e3a1c4a5b44340be8a183d5d649b27ac
I20240501 08:13:54.134038 7634 raft_consensus.cc:2781] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:54.134199 7634 raft_consensus.cc:686] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:13:54.134510 7634 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.137216 6496 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:13:54.137553 7635 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:54.137737 7635 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.139676 7635 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:13:54.147262 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070e927b5d684367ae0b40b12a23da54" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:13:54.147310 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070e927b5d684367ae0b40b12a23da54" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:13:54.147539 6769 raft_consensus.cc:2454] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:13:54.147558 6898 raft_consensus.cc:2454] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:13:54.148067 6571 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:54.148496 7635 raft_consensus.cc:2781] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:54.148628 7635 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:54.148711 7635 raft_consensus.cc:3037] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:54.150914 7635 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.151624 7635 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:13:54.152195 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070e927b5d684367ae0b40b12a23da54" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:13:54.152235 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070e927b5d684367ae0b40b12a23da54" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:13:54.152451 6769 raft_consensus.cc:3037] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:54.152415 6898 raft_consensus.cc:3037] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:54.154484 6769 raft_consensus.cc:2454] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:13:54.154742 6898 raft_consensus.cc:2454] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:13:54.155005 6574 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:13:54.155350 7635 raft_consensus.cc:2781] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:54.155766 7635 raft_consensus.cc:686] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:13:54.156302 7635 consensus_queue.cc:237] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:54.163409 6496 catalog_manager.cc:5484] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:13:54.521087 6898 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:54.525493 7634 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:54.535296 7634 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:54.557757 7652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:54.558137 7652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:54.565861 7652 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 08:13:54.645010 7634 consensus_queue.cc:1035] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:54.651893 7645 consensus_queue.cc:1035] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:54.656090 7635 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:54.690759 7635 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:54.807675 6962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.3:43505, user_credentials={real_user=slave}} blocked reactor thread for 49511.9us
INSERT report
rows total: 2000
time total: 242.748 ms
time per row: 0.121374 ms
I20240501 08:13:54.962823 6495 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42016:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:54.972844 6749 tablet_service.cc:1467] Processing CreateTablet for tablet a9e70954aa4e4fcfbad12e8bfe3e0059 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:54.973004 6878 tablet_service.cc:1467] Processing CreateTablet for tablet a9e70954aa4e4fcfbad12e8bfe3e0059 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:54.973424 6749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e70954aa4e4fcfbad12e8bfe3e0059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:54.973489 6878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e70954aa4e4fcfbad12e8bfe3e0059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:54.973484 6748 tablet_service.cc:1467] Processing CreateTablet for tablet 193de3c11ae54408a64ae3007bf88920 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:54.973853 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193de3c11ae54408a64ae3007bf88920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:54.974738 7009 tablet_service.cc:1467] Processing CreateTablet for tablet a9e70954aa4e4fcfbad12e8bfe3e0059 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:54.975176 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e70954aa4e4fcfbad12e8bfe3e0059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:54.975582 6879 tablet_service.cc:1467] Processing CreateTablet for tablet 193de3c11ae54408a64ae3007bf88920 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:54.976016 6879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193de3c11ae54408a64ae3007bf88920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:54.977208 6877 tablet_service.cc:1467] Processing CreateTablet for tablet 4db7f887af8f425c8fa661196add4d37 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:54.977792 6877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db7f887af8f425c8fa661196add4d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:54.980859 6617 tablet_service.cc:1467] Processing CreateTablet for tablet 4db7f887af8f425c8fa661196add4d37 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:54.981483 6617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db7f887af8f425c8fa661196add4d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:54.981796 7008 tablet_service.cc:1467] Processing CreateTablet for tablet 193de3c11ae54408a64ae3007bf88920 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:54.982220 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193de3c11ae54408a64ae3007bf88920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:54.984510 6747 tablet_service.cc:1467] Processing CreateTablet for tablet 4db7f887af8f425c8fa661196add4d37 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:54.985055 6747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db7f887af8f425c8fa661196add4d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:54.994849 7712 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:54.997318 7712 tablet_bootstrap.cc:654] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:54.997558 7714 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:54.999608 7715 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:55.001097 7713 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:55.001163 7714 tablet_bootstrap.cc:654] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:55.002151 7715 tablet_bootstrap.cc:654] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:55.003147 7713 tablet_bootstrap.cc:654] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:55.003536 7712 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:55.003702 7712 ts_tablet_manager.cc:1367] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 08:13:55.004539 7712 raft_consensus.cc:348] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:55.004776 7712 raft_consensus.cc:374] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:55.004874 7712 raft_consensus.cc:729] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:55.005133 7712 consensus_queue.cc:260] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:55.005380 7713 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:55.005730 7712 ts_tablet_manager.cc:1398] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:55.005817 7714 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:55.005952 7715 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:55.006000 7714 ts_tablet_manager.cc:1367] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240501 08:13:55.006076 7712 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:55.006188 7713 ts_tablet_manager.cc:1367] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:13:55.006219 7715 ts_tablet_manager.cc:1367] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:13:55.007236 7714 raft_consensus.cc:348] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.007454 7714 raft_consensus.cc:374] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:55.007561 7714 raft_consensus.cc:729] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:55.007598 7715 raft_consensus.cc:348] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:55.007900 7715 raft_consensus.cc:374] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:55.007931 7712 tablet_bootstrap.cc:654] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:55.007810 7714 consensus_queue.cc:260] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.008424 7714 ts_tablet_manager.cc:1398] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:55.008877 7715 raft_consensus.cc:729] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:55.009213 7715 consensus_queue.cc:260] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:55.009755 7713 raft_consensus.cc:348] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:55.010016 7713 raft_consensus.cc:374] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:55.010102 7713 raft_consensus.cc:729] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:55.010417 7713 consensus_queue.cc:260] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:55.011008 7713 ts_tablet_manager.cc:1398] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:13:55.011335 7713 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:55.012449 7715 ts_tablet_manager.cc:1398] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:13:55.019928 7715 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:55.021857 7715 tablet_bootstrap.cc:654] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:55.023038 7713 tablet_bootstrap.cc:654] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:55.030019 7713 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:55.030315 7713 ts_tablet_manager.cc:1367] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 08:13:55.031265 7713 raft_consensus.cc:348] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.031523 7713 raft_consensus.cc:374] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:55.031605 7713 raft_consensus.cc:729] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:55.031929 7713 consensus_queue.cc:260] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.032629 7713 ts_tablet_manager.cc:1398] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:13:55.034246 7715 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:55.034565 7715 ts_tablet_manager.cc:1367] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20240501 08:13:55.036509 7715 raft_consensus.cc:348] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.036861 7715 raft_consensus.cc:374] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:55.035557 7712 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:55.037271 7712 ts_tablet_manager.cc:1367] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.016s
I20240501 08:13:55.037412 7715 raft_consensus.cc:729] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:55.037827 7715 consensus_queue.cc:260] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.038671 7715 ts_tablet_manager.cc:1398] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:55.039088 7715 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:55.040832 7715 tablet_bootstrap.cc:654] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:55.046241 7712 raft_consensus.cc:348] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.046618 7712 raft_consensus.cc:374] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:55.046707 7712 raft_consensus.cc:729] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:55.047042 7712 consensus_queue.cc:260] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.049521 7715 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:55.049787 7715 ts_tablet_manager.cc:1367] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240501 08:13:55.050621 7712 ts_tablet_manager.cc:1398] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 08:13:55.050837 7715 raft_consensus.cc:348] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.051215 7715 raft_consensus.cc:374] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:55.051373 7715 raft_consensus.cc:729] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:55.051687 7715 consensus_queue.cc:260] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.052402 7712 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:55.052628 7715 ts_tablet_manager.cc:1398] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:55.054101 7712 tablet_bootstrap.cc:654] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:55.056689 7712 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:55.056854 7712 ts_tablet_manager.cc:1367] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:13:55.057531 7712 raft_consensus.cc:348] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.057715 7712 raft_consensus.cc:374] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:55.057775 7712 raft_consensus.cc:729] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:55.058025 7712 consensus_queue.cc:260] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.058591 7712 ts_tablet_manager.cc:1398] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:55.150162 7703 raft_consensus.cc:482] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:55.150609 7703 raft_consensus.cc:504] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:55.151424 7703 leader_election.cc:288] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:55.151988 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e70954aa4e4fcfbad12e8bfe3e0059" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:13:55.152020 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e70954aa4e4fcfbad12e8bfe3e0059" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:13:55.152241 6769 raft_consensus.cc:2454] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:13:55.152276 6899 raft_consensus.cc:2454] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:13:55.152765 6962 leader_election.cc:302] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:55.153105 7703 raft_consensus.cc:2781] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:55.153229 7703 raft_consensus.cc:482] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:55.153301 7703 raft_consensus.cc:3037] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:55.155284 7703 raft_consensus.cc:504] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:55.155934 7703 leader_election.cc:288] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:55.156512 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e70954aa4e4fcfbad12e8bfe3e0059" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:13:55.156558 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e70954aa4e4fcfbad12e8bfe3e0059" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:13:55.156764 6899 raft_consensus.cc:3037] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:55.156764 6769 raft_consensus.cc:3037] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:55.159730 6769 raft_consensus.cc:2454] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:13:55.159730 6899 raft_consensus.cc:2454] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:13:55.160251 6965 leader_election.cc:302] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:13:55.160622 7703 raft_consensus.cc:2781] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:55.160784 7703 raft_consensus.cc:686] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:13:55.161152 7703 consensus_queue.cc:237] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:55.164237 6497 catalog_manager.cc:5484] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:13:55.242734 7681 raft_consensus.cc:482] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:55.243048 7681 raft_consensus.cc:504] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.245169 7681 leader_election.cc:288] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:55.253623 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193de3c11ae54408a64ae3007bf88920" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:13:55.253898 6899 raft_consensus.cc:2454] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:13:55.253724 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193de3c11ae54408a64ae3007bf88920" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:13:55.254145 7029 raft_consensus.cc:2454] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:13:55.254411 6702 leader_election.cc:302] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:55.254824 7681 raft_consensus.cc:2781] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:55.254947 7681 raft_consensus.cc:482] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:55.255028 7681 raft_consensus.cc:3037] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:55.257061 7681 raft_consensus.cc:504] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.257728 7681 leader_election.cc:288] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:55.258435 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193de3c11ae54408a64ae3007bf88920" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:13:55.258443 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193de3c11ae54408a64ae3007bf88920" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:13:55.258659 7029 raft_consensus.cc:3037] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:55.258679 6899 raft_consensus.cc:3037] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:55.261586 6899 raft_consensus.cc:2454] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:13:55.261595 7029 raft_consensus.cc:2454] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:13:55.262135 6702 leader_election.cc:302] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:55.262526 7681 raft_consensus.cc:2781] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:55.262951 7681 raft_consensus.cc:686] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Becoming Leader. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:13:55.263465 7681 consensus_queue.cc:237] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.266799 6497 catalog_manager.cc:5484] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2). New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:13:55.398360 7650 raft_consensus.cc:482] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:55.398542 7650 raft_consensus.cc:504] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.399250 7650 leader_election.cc:288] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:55.399878 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db7f887af8f425c8fa661196add4d37" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:13:55.399924 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db7f887af8f425c8fa661196add4d37" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:13:55.400131 6769 raft_consensus.cc:2454] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:13:55.400173 6899 raft_consensus.cc:2454] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:13:55.400615 6574 leader_election.cc:302] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:13:55.401026 7650 raft_consensus.cc:2781] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:55.401154 7650 raft_consensus.cc:482] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:55.401235 7650 raft_consensus.cc:3037] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:55.404177 7650 raft_consensus.cc:504] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.404973 7650 leader_election.cc:288] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:55.405543 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db7f887af8f425c8fa661196add4d37" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:13:55.405588 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db7f887af8f425c8fa661196add4d37" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:13:55.405767 6769 raft_consensus.cc:3037] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:55.405824 6898 raft_consensus.cc:3037] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:55.408478 6898 raft_consensus.cc:2454] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:13:55.408586 6769 raft_consensus.cc:2454] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:13:55.408936 6571 leader_election.cc:302] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:55.409263 7650 raft_consensus.cc:2781] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:55.409636 7650 raft_consensus.cc:686] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:13:55.409943 7650 consensus_queue.cc:237] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:55.412617 6497 catalog_manager.cc:5484] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:13:55.548137 7705 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:55.563113 7705 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:55.690030 7681 consensus_queue.cc:1035] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:55.695982 7681 consensus_queue.cc:1035] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:55.845374 7733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:55.845742 7733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:55.853682 7733 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:13:55.908140 7650 consensus_queue.cc:1035] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:55.963680 7650 consensus_queue.cc:1035] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 281.704 ms
time per row: 0.140852 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:13:57.159853 6491 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42016:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:57.169425 6617 tablet_service.cc:1467] Processing CreateTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:57.169839 7009 tablet_service.cc:1467] Processing CreateTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:57.169946 6617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4f075057e42d0bb3f8cdabdfbc643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:57.170243 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4f075057e42d0bb3f8cdabdfbc643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:57.170647 6747 tablet_service.cc:1467] Processing CreateTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:57.171097 6747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4f075057e42d0bb3f8cdabdfbc643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:57.173439 7007 tablet_service.cc:1467] Processing CreateTablet for tablet fdaad8e1d86942ddbc97958dcf31b344 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:57.173879 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdaad8e1d86942ddbc97958dcf31b344. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:57.177191 6618 tablet_service.cc:1467] Processing CreateTablet for tablet 78d1df7b0ea44f89b9f22c156f907c1e (DEFAULT_TABLE table=rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:57.177680 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d1df7b0ea44f89b9f22c156f907c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:57.180495 7810 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:57.182746 6877 tablet_service.cc:1467] Processing CreateTablet for tablet 78d1df7b0ea44f89b9f22c156f907c1e (DEFAULT_TABLE table=rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:57.183260 6877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d1df7b0ea44f89b9f22c156f907c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:57.183293 7809 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:57.184830 7809 tablet_bootstrap.cc:654] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:57.186371 7008 tablet_service.cc:1467] Processing CreateTablet for tablet 78d1df7b0ea44f89b9f22c156f907c1e (DEFAULT_TABLE table=rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:57.186848 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d1df7b0ea44f89b9f22c156f907c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:57.190465 7812 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:57.190745 6616 tablet_service.cc:1467] Processing CreateTablet for tablet fdaad8e1d86942ddbc97958dcf31b344 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:57.191558 7809 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:57.191780 7809 ts_tablet_manager.cc:1367] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240501 08:13:57.192176 7812 tablet_bootstrap.cc:654] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:57.192220 6748 tablet_service.cc:1467] Processing CreateTablet for tablet fdaad8e1d86942ddbc97958dcf31b344 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:57.192701 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdaad8e1d86942ddbc97958dcf31b344. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:57.192705 7809 raft_consensus.cc:348] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.192986 7809 raft_consensus.cc:374] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:57.193084 7809 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:57.193383 7809 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.193985 7809 ts_tablet_manager.cc:1398] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:57.194346 7809 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:57.195134 7810 tablet_bootstrap.cc:654] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:57.197906 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdaad8e1d86942ddbc97958dcf31b344. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:57.198849 7814 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:57.203438 7814 tablet_bootstrap.cc:654] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:57.206053 7810 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:57.206248 7810 ts_tablet_manager.cc:1367] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.000s
I20240501 08:13:57.206444 7812 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:57.206650 7812 ts_tablet_manager.cc:1367] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20240501 08:13:57.206876 7809 tablet_bootstrap.cc:654] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:57.206931 7814 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:57.207072 7814 ts_tablet_manager.cc:1367] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 08:13:57.207218 7810 raft_consensus.cc:348] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.207528 7810 raft_consensus.cc:374] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:57.207674 7810 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:57.208015 7810 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.208112 7814 raft_consensus.cc:348] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.208456 7812 raft_consensus.cc:348] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:57.208741 7812 raft_consensus.cc:374] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:57.208880 7812 raft_consensus.cc:729] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:57.209173 7812 consensus_queue.cc:260] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:57.209689 7814 raft_consensus.cc:374] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:57.209885 7814 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:57.210242 7814 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.211107 7814 ts_tablet_manager.cc:1398] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:13:57.212131 7814 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:57.214287 7814 tablet_bootstrap.cc:654] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:57.218369 7810 ts_tablet_manager.cc:1398] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 08:13:57.218828 7810 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:57.219065 7814 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:57.219286 7814 ts_tablet_manager.cc:1367] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 08:13:57.219489 7812 ts_tablet_manager.cc:1398] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.013s user 0.006s sys 0.004s
I20240501 08:13:57.220129 7814 raft_consensus.cc:348] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.220383 7809 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:57.220418 7814 raft_consensus.cc:374] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:57.220494 7814 raft_consensus.cc:729] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:57.220629 7809 ts_tablet_manager.cc:1367] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.001s
I20240501 08:13:57.220773 7814 consensus_queue.cc:260] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.221344 7814 ts_tablet_manager.cc:1398] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:13:57.221524 7810 tablet_bootstrap.cc:654] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:57.222147 7809 raft_consensus.cc:348] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.222492 7809 raft_consensus.cc:374] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:57.222626 7809 raft_consensus.cc:729] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:57.222992 7809 consensus_queue.cc:260] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.223639 7809 ts_tablet_manager.cc:1398] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:57.224038 7809 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:57.225872 7809 tablet_bootstrap.cc:654] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:57.226147 7810 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:57.226430 7810 ts_tablet_manager.cc:1367] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:13:57.227365 7810 raft_consensus.cc:348] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:57.227743 7810 raft_consensus.cc:374] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:57.227835 7810 raft_consensus.cc:729] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:57.228155 7810 consensus_queue.cc:260] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:57.229528 7810 ts_tablet_manager.cc:1398] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:57.229952 7810 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:57.231948 7810 tablet_bootstrap.cc:654] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:57.234081 7809 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:57.234305 7809 ts_tablet_manager.cc:1367] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 08:13:57.234077 7810 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:57.234491 7810 ts_tablet_manager.cc:1367] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:13:57.235127 7809 raft_consensus.cc:348] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:57.235163 7810 raft_consensus.cc:348] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.235347 7809 raft_consensus.cc:374] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:57.235347 7810 raft_consensus.cc:374] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:57.235445 7809 raft_consensus.cc:729] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:57.235452 7810 raft_consensus.cc:729] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:57.235679 7810 consensus_queue.cc:260] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.235679 7809 consensus_queue.cc:260] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:57.236225 7810 ts_tablet_manager.cc:1398] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:57.236240 7809 ts_tablet_manager.cc:1398] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:57.279978 7804 raft_consensus.cc:482] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:57.280185 7804 raft_consensus.cc:504] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.280978 7804 leader_election.cc:288] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:57.281666 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdaad8e1d86942ddbc97958dcf31b344" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:13:57.281994 7029 raft_consensus.cc:2454] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:13:57.282552 6704 leader_election.cc:302] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:13:57.282902 7804 raft_consensus.cc:2781] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:57.283038 7804 raft_consensus.cc:482] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:57.283164 7804 raft_consensus.cc:3037] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:57.285998 7804 raft_consensus.cc:504] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.286702 7804 leader_election.cc:288] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:57.287259 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdaad8e1d86942ddbc97958dcf31b344" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:13:57.287516 7029 raft_consensus.cc:3037] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:57.289350 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdaad8e1d86942ddbc97958dcf31b344" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:13:57.289388 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdaad8e1d86942ddbc97958dcf31b344" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:13:57.289623 6637 raft_consensus.cc:2454] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:13:57.290333 7029 raft_consensus.cc:2454] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:13:57.290784 6704 leader_election.cc:302] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, cd3434d87b3c46c6848a711cdf6b150c; no voters: 97cdee0bf9d84645a1451f9822c5d699
I20240501 08:13:57.291138 7804 raft_consensus.cc:2781] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:57.291617 7804 raft_consensus.cc:686] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Becoming Leader. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:13:57.291921 7804 consensus_queue.cc:237] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.294724 6488 catalog_manager.cc:5484] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2). New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:13:57.312502 7766 raft_consensus.cc:482] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:57.312711 7766 raft_consensus.cc:504] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.313517 7766 leader_election.cc:288] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:13:57.314244 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:13:57.314551 6769 raft_consensus.cc:2454] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:13:57.315064 6574 leader_election.cc:302] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:13:57.315438 7766 raft_consensus.cc:2781] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:57.315577 7766 raft_consensus.cc:482] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:57.315665 7766 raft_consensus.cc:3037] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:57.317620 7766 raft_consensus.cc:504] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.318390 7766 leader_election.cc:288] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:13:57.319104 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:13:57.319329 6769 raft_consensus.cc:3037] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:57.320394 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:13:57.320442 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:13:57.320703 7029 raft_consensus.cc:2454] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:13:57.321779 6769 raft_consensus.cc:2454] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:13:57.322366 6574 leader_election.cc:302] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c; no voters: 28b7613bd4834f3ab245c82dd7e80240
I20240501 08:13:57.323091 7766 raft_consensus.cc:2781] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:57.323499 7766 raft_consensus.cc:686] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:13:57.323812 7766 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:57.327059 6488 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:13:57.463541 7766 raft_consensus.cc:482] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:57.463778 7766 raft_consensus.cc:504] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:57.464505 7766 leader_election.cc:288] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:57.465162 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d1df7b0ea44f89b9f22c156f907c1e" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:13:57.465183 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d1df7b0ea44f89b9f22c156f907c1e" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:13:57.465406 7029 raft_consensus.cc:2454] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:13:57.465399 6899 raft_consensus.cc:2454] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:13:57.465891 6573 leader_election.cc:302] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:13:57.466192 7766 raft_consensus.cc:2781] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:57.466312 7766 raft_consensus.cc:482] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:57.466410 7766 raft_consensus.cc:3037] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:57.468313 7766 raft_consensus.cc:504] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:57.468953 7766 leader_election.cc:288] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:57.469544 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d1df7b0ea44f89b9f22c156f907c1e" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:13:57.469544 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d1df7b0ea44f89b9f22c156f907c1e" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:13:57.469781 7029 raft_consensus.cc:3037] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:57.469785 6899 raft_consensus.cc:3037] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:57.473407 7029 raft_consensus.cc:2454] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:13:57.473407 6899 raft_consensus.cc:2454] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:13:57.474017 6573 leader_election.cc:302] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:13:57.474377 7766 raft_consensus.cc:2781] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:57.474551 7766 raft_consensus.cc:686] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:13:57.474822 7766 consensus_queue.cc:237] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:57.477910 6488 catalog_manager.cc:5484] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:13:57.838984 7029 raft_consensus.cc:3037] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:57.842015 7766 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:57.852133 7804 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:57.860492 7766 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:57.872781 6638 raft_consensus.cc:3037] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:57.877002 7804 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:57.961190 7832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:57.961547 7832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:57.970311 7832 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 08:13:58.009513 7829 consensus_queue.cc:1035] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:58.015729 7766 consensus_queue.cc:1035] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 219.702 ms
time per row: 0.109851 ms
I20240501 08:13:58.364121 6488 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42016:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:58.374192 6877 tablet_service.cc:1467] Processing CreateTablet for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:58.374410 6748 tablet_service.cc:1467] Processing CreateTablet for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:58.374737 6877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:58.374950 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:58.375074 6747 tablet_service.cc:1467] Processing CreateTablet for tablet 57ecbb30f7764fceb008baaa2c37da8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:58.375468 6747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ecbb30f7764fceb008baaa2c37da8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:58.380641 7901 tablet_bootstrap.cc:492] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:58.381359 6616 tablet_service.cc:1467] Processing CreateTablet for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:58.381353 6879 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6efa669c1645aaa971d59e5d38c8d5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:58.381498 6878 tablet_service.cc:1467] Processing CreateTablet for tablet 57ecbb30f7764fceb008baaa2c37da8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:58.381796 6879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6efa669c1645aaa971d59e5d38c8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:58.381791 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:58.382287 6618 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6efa669c1645aaa971d59e5d38c8d5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:58.382678 7901 tablet_bootstrap.cc:654] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:58.382714 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6efa669c1645aaa971d59e5d38c8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:58.383301 7008 tablet_service.cc:1467] Processing CreateTablet for tablet 57ecbb30f7764fceb008baaa2c37da8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:58.383730 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ecbb30f7764fceb008baaa2c37da8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:58.386778 7007 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6efa669c1645aaa971d59e5d38c8d5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:58.387259 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6efa669c1645aaa971d59e5d38c8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:58.389567 6878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ecbb30f7764fceb008baaa2c37da8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:58.393105 7903 tablet_bootstrap.cc:492] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:58.395522 7903 tablet_bootstrap.cc:654] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:58.404742 7905 tablet_bootstrap.cc:492] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:58.408066 7905 tablet_bootstrap.cc:654] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:58.418193 7905 tablet_bootstrap.cc:492] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:58.418607 7905 ts_tablet_manager.cc:1367] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240501 08:13:58.419747 7905 raft_consensus.cc:348] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:58.420192 7905 raft_consensus.cc:374] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:58.420343 7905 raft_consensus.cc:729] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:58.420732 7905 consensus_queue.cc:260] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:58.421653 7905 ts_tablet_manager.cc:1398] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:58.432049 7905 tablet_bootstrap.cc:492] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:58.435086 7906 tablet_bootstrap.cc:492] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:58.436417 7905 tablet_bootstrap.cc:654] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:58.437811 7906 tablet_bootstrap.cc:654] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:58.446383 7905 tablet_bootstrap.cc:492] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:58.446713 7905 ts_tablet_manager.cc:1367] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.001s
I20240501 08:13:58.447337 7903 tablet_bootstrap.cc:492] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:58.447337 7906 tablet_bootstrap.cc:492] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:58.447520 7903 ts_tablet_manager.cc:1367] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.028s
I20240501 08:13:58.447541 7906 ts_tablet_manager.cc:1367] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20240501 08:13:58.447471 7905 raft_consensus.cc:348] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:58.447728 7905 raft_consensus.cc:374] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:58.447803 7905 raft_consensus.cc:729] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:58.448073 7905 consensus_queue.cc:260] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:58.448351 7903 raft_consensus.cc:348] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:58.448563 7903 raft_consensus.cc:374] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:58.448614 7905 ts_tablet_manager.cc:1398] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:58.448663 7903 raft_consensus.cc:729] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:58.448945 7903 consensus_queue.cc:260] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:58.448350 7906 raft_consensus.cc:348] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:58.449414 7906 raft_consensus.cc:374] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:58.449506 7906 raft_consensus.cc:729] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:58.449515 7903 ts_tablet_manager.cc:1398] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:58.449805 7903 tablet_bootstrap.cc:492] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:58.449769 7906 consensus_queue.cc:260] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:58.450340 7906 ts_tablet_manager.cc:1398] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 08:13:58.451220 7906 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:58.452819 7903 tablet_bootstrap.cc:654] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:58.453488 7906 tablet_bootstrap.cc:654] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:58.457376 7901 tablet_bootstrap.cc:492] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:58.457579 7901 ts_tablet_manager.cc:1367] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.077s user 0.038s sys 0.023s
I20240501 08:13:58.458696 7901 raft_consensus.cc:348] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:58.458928 7901 raft_consensus.cc:374] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:58.459012 7901 raft_consensus.cc:729] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:58.459273 7901 consensus_queue.cc:260] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:58.459421 7903 tablet_bootstrap.cc:492] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:58.459619 7903 ts_tablet_manager.cc:1367] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 08:13:58.459870 7901 ts_tablet_manager.cc:1398] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:58.460179 7901 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:58.460726 7903 raft_consensus.cc:348] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:58.460995 7903 raft_consensus.cc:374] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:58.461098 7903 raft_consensus.cc:729] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:58.461374 7903 consensus_queue.cc:260] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:58.461840 7901 tablet_bootstrap.cc:654] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:58.462131 7903 ts_tablet_manager.cc:1398] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:58.462507 7903 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:58.464319 7906 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:58.464313 7901 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:58.464485 7901 ts_tablet_manager.cc:1367] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:13:58.464572 7903 tablet_bootstrap.cc:654] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:58.464603 7906 ts_tablet_manager.cc:1367] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20240501 08:13:58.465463 7906 raft_consensus.cc:348] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:58.465757 7906 raft_consensus.cc:374] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:58.465883 7906 raft_consensus.cc:729] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:58.466228 7906 consensus_queue.cc:260] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:58.466553 7901 raft_consensus.cc:348] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:58.466852 7901 raft_consensus.cc:374] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:58.466992 7901 raft_consensus.cc:729] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:58.467136 7906 ts_tablet_manager.cc:1398] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:58.467468 7901 consensus_queue.cc:260] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:58.468914 7903 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:58.468992 7901 ts_tablet_manager.cc:1398] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:13:58.469156 7903 ts_tablet_manager.cc:1367] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240501 08:13:58.470005 7903 raft_consensus.cc:348] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:58.470219 7903 raft_consensus.cc:374] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:58.470383 7903 raft_consensus.cc:729] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:58.470638 7903 consensus_queue.cc:260] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:58.471349 7903 ts_tablet_manager.cc:1398] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:58.483755 7703 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:58.484010 7703 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:58.484757 7703 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:58.485416 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:13:58.485415 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:13:58.485687 6899 raft_consensus.cc:2454] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:13:58.485687 6769 raft_consensus.cc:2454] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:13:58.486209 6962 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:58.486560 7703 raft_consensus.cc:2781] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:58.486640 7703 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:58.486728 7703 raft_consensus.cc:3037] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:58.488622 7703 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:58.489264 7703 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:58.490110 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:13:58.490221 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:13:58.490481 6899 raft_consensus.cc:3037] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:58.490541 6769 raft_consensus.cc:3037] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:58.492517 6769 raft_consensus.cc:2454] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:13:58.492621 6899 raft_consensus.cc:2454] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:13:58.493131 6965 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:13:58.493463 7703 raft_consensus.cc:2781] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:58.493872 7703 raft_consensus.cc:686] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:13:58.494163 7703 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:13:58.497005 6488 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:13:58.535677 7841 raft_consensus.cc:482] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:58.535893 7841 raft_consensus.cc:504] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:58.536571 7841 leader_election.cc:288] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:58.537192 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358b6fe9c68349cb8cb963bb3a9ed2e9" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:13:58.537238 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358b6fe9c68349cb8cb963bb3a9ed2e9" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:13:58.537452 6638 raft_consensus.cc:2454] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:13:58.537451 6899 raft_consensus.cc:2454] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:13:58.537910 6702 leader_election.cc:302] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:13:58.538192 7841 raft_consensus.cc:2781] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:58.538329 7841 raft_consensus.cc:482] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:58.538409 7841 raft_consensus.cc:3037] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:58.540357 7841 raft_consensus.cc:504] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:58.540923 7841 leader_election.cc:288] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Requested vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:58.541570 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358b6fe9c68349cb8cb963bb3a9ed2e9" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:13:58.541546 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358b6fe9c68349cb8cb963bb3a9ed2e9" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:13:58.541759 6638 raft_consensus.cc:3037] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:58.541831 6899 raft_consensus.cc:3037] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:58.543769 6638 raft_consensus.cc:2454] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:13:58.543857 6899 raft_consensus.cc:2454] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:13:58.544301 6702 leader_election.cc:302] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:13:58.544687 7841 raft_consensus.cc:2781] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:58.544828 7841 raft_consensus.cc:686] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Becoming Leader. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:13:58.545151 7841 consensus_queue.cc:237] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:58.546883 7703 raft_consensus.cc:482] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:58.547048 7703 raft_consensus.cc:504] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:58.547731 7703 leader_election.cc:288] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:58.548296 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6efa669c1645aaa971d59e5d38c8d5" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:13:58.548350 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6efa669c1645aaa971d59e5d38c8d5" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:13:58.548571 6899 raft_consensus.cc:2454] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:13:58.548571 6638 raft_consensus.cc:2454] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:13:58.548388 6488 catalog_manager.cc:5484] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2). New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:13:58.549057 6962 leader_election.cc:302] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:58.549373 7703 raft_consensus.cc:2781] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:58.549503 7703 raft_consensus.cc:482] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:58.549600 7703 raft_consensus.cc:3037] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:58.552166 7703 raft_consensus.cc:504] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:58.552809 7703 leader_election.cc:288] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Requested vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:58.553333 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6efa669c1645aaa971d59e5d38c8d5" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:13:58.553591 6899 raft_consensus.cc:3037] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:58.553529 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6efa669c1645aaa971d59e5d38c8d5" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:13:58.553746 6638 raft_consensus.cc:3037] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:58.556464 6638 raft_consensus.cc:2454] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:13:58.556464 6899 raft_consensus.cc:2454] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:13:58.557075 6962 leader_election.cc:302] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:13:58.557446 7703 raft_consensus.cc:2781] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:58.557629 7703 raft_consensus.cc:686] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:13:58.557971 7703 consensus_queue.cc:237] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:13:58.560997 6488 catalog_manager.cc:5484] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:13:58.904381 7703 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:58.911118 7703 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:58.964579 7841 consensus_queue.cc:1035] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:58.969528 7842 consensus_queue.cc:1035] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:58.991264 7918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:58.991629 7918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:58.999723 7918 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 08:13:59.120101 7914 consensus_queue.cc:1035] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:59.126514 7914 consensus_queue.cc:1035] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 241.645 ms
time per row: 0.120823 ms
I20240501 08:13:59.413829 6492 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42016:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:59.424772 6747 tablet_service.cc:1467] Processing CreateTablet for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:59.424809 6878 tablet_service.cc:1467] Processing CreateTablet for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:59.425347 6747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:59.425478 6878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:59.426303 6748 tablet_service.cc:1467] Processing CreateTablet for tablet dc19d80bad46460087c143778a4b0638 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:59.426775 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc19d80bad46460087c143778a4b0638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:59.430413 6616 tablet_service.cc:1467] Processing CreateTablet for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:59.430408 7007 tablet_service.cc:1467] Processing CreateTablet for tablet dc19d80bad46460087c143778a4b0638 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:59.430895 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:59.430907 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc19d80bad46460087c143778a4b0638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:59.431126 6879 tablet_service.cc:1467] Processing CreateTablet for tablet a7a56c284444496b8579600a49f19d1d (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:59.431279 6618 tablet_service.cc:1467] Processing CreateTablet for tablet dc19d80bad46460087c143778a4b0638 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:59.431597 6879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a56c284444496b8579600a49f19d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:59.431679 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc19d80bad46460087c143778a4b0638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:59.437762 6749 tablet_service.cc:1467] Processing CreateTablet for tablet a7a56c284444496b8579600a49f19d1d (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:59.438349 6749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a56c284444496b8579600a49f19d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:59.442844 7008 tablet_service.cc:1467] Processing CreateTablet for tablet a7a56c284444496b8579600a49f19d1d (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:59.443377 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a56c284444496b8579600a49f19d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:59.445520 7983 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:59.447748 7983 tablet_bootstrap.cc:654] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:59.452514 7986 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:59.456176 7986 tablet_bootstrap.cc:654] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:59.457770 7985 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:59.457939 7983 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:59.458137 7983 ts_tablet_manager.cc:1367] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240501 08:13:59.458874 7984 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:59.459075 7983 raft_consensus.cc:348] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.459295 7983 raft_consensus.cc:374] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:59.459375 7983 raft_consensus.cc:729] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:59.459625 7983 consensus_queue.cc:260] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.460203 7983 ts_tablet_manager.cc:1398] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:59.460523 7983 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:13:59.460637 7984 tablet_bootstrap.cc:654] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:59.460294 7986 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:59.461570 7986 ts_tablet_manager.cc:1367] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 08:13:59.462317 7983 tablet_bootstrap.cc:654] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:59.462520 7986 raft_consensus.cc:348] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.462749 7986 raft_consensus.cc:374] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:59.462889 7986 raft_consensus.cc:729] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:59.463220 7986 consensus_queue.cc:260] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.464146 7986 ts_tablet_manager.cc:1398] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:59.464502 7986 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:59.465636 7985 tablet_bootstrap.cc:654] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:59.466230 7986 tablet_bootstrap.cc:654] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:59.466764 7983 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:13:59.466970 7983 ts_tablet_manager.cc:1367] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:13:59.467871 7983 raft_consensus.cc:348] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.468142 7983 raft_consensus.cc:374] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:59.468221 7983 raft_consensus.cc:729] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:13:59.468508 7983 consensus_queue.cc:260] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.471822 7983 ts_tablet_manager.cc:1398] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:59.472605 7984 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:59.472774 7984 ts_tablet_manager.cc:1367] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.006s
I20240501 08:13:59.473654 7984 raft_consensus.cc:348] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:59.473826 7985 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:59.473898 7984 raft_consensus.cc:374] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:59.473974 7985 ts_tablet_manager.cc:1367] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20240501 08:13:59.474000 7984 raft_consensus.cc:729] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:59.474193 7986 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:59.474304 7984 consensus_queue.cc:260] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:59.474458 7986 ts_tablet_manager.cc:1367] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20240501 08:13:59.474841 7985 raft_consensus.cc:348] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:59.475070 7985 raft_consensus.cc:374] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:59.475170 7985 raft_consensus.cc:729] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:59.475440 7985 consensus_queue.cc:260] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:59.475625 7984 ts_tablet_manager.cc:1398] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:13:59.475802 7986 raft_consensus.cc:348] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.475960 7984 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:13:59.476054 7985 ts_tablet_manager.cc:1398] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:59.476111 7986 raft_consensus.cc:374] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:59.476266 7986 raft_consensus.cc:729] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:59.476369 7985 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:13:59.476608 7986 consensus_queue.cc:260] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.477782 7984 tablet_bootstrap.cc:654] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:59.478132 7985 tablet_bootstrap.cc:654] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:59.480783 7986 ts_tablet_manager.cc:1398] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:13:59.481284 7986 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:13:59.484755 7986 tablet_bootstrap.cc:654] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:59.485405 7985 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:13:59.485639 7985 ts_tablet_manager.cc:1367] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20240501 08:13:59.487107 7985 raft_consensus.cc:348] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.487442 7985 raft_consensus.cc:374] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:59.487555 7985 raft_consensus.cc:729] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:13:59.487895 7985 consensus_queue.cc:260] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.489109 7985 ts_tablet_manager.cc:1398] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:59.489774 7984 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:13:59.490000 7984 ts_tablet_manager.cc:1367] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20240501 08:13:59.490202 7986 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:13:59.490387 7986 ts_tablet_manager.cc:1367] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240501 08:13:59.490968 7984 raft_consensus.cc:348] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.491218 7984 raft_consensus.cc:374] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:59.491294 7984 raft_consensus.cc:729] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:13:59.491180 7986 raft_consensus.cc:348] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:59.491575 7986 raft_consensus.cc:374] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:59.491559 7984 consensus_queue.cc:260] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.491683 7986 raft_consensus.cc:729] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:13:59.491910 7986 consensus_queue.cc:260] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:59.492172 7984 ts_tablet_manager.cc:1398] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:13:59.492509 7986 ts_tablet_manager.cc:1398] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:59.535862 7842 raft_consensus.cc:482] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:59.536094 7842 raft_consensus.cc:504] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.536718 7842 leader_election.cc:288] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:59.537376 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a56c284444496b8579600a49f19d1d" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:13:59.537367 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a56c284444496b8579600a49f19d1d" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:13:59.537616 6899 raft_consensus.cc:2454] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:13:59.537614 7029 raft_consensus.cc:2454] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:13:59.538147 6702 leader_election.cc:302] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:59.538544 7842 raft_consensus.cc:2781] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:59.538669 7842 raft_consensus.cc:482] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:59.538771 7842 raft_consensus.cc:3037] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:59.538735 7881 raft_consensus.cc:482] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:59.538900 7881 raft_consensus.cc:504] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.539685 7881 leader_election.cc:288] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:13:59.540202 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19d80bad46460087c143778a4b0638" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:13:59.540205 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19d80bad46460087c143778a4b0638" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:13:59.540437 7029 raft_consensus.cc:2454] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:13:59.540449 6769 raft_consensus.cc:2454] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:13:59.540931 6573 leader_election.cc:302] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:13:59.541225 7881 raft_consensus.cc:2781] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:59.541345 7881 raft_consensus.cc:482] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:59.541451 7881 raft_consensus.cc:3037] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:59.541518 7842 raft_consensus.cc:504] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.542177 7842 leader_election.cc:288] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:13:59.542629 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a56c284444496b8579600a49f19d1d" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:13:59.542834 7029 raft_consensus.cc:3037] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:59.542805 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a56c284444496b8579600a49f19d1d" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:13:59.543005 6899 raft_consensus.cc:3037] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:59.544055 7881 raft_consensus.cc:504] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.544821 7881 leader_election.cc:288] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:13:59.545064 6899 raft_consensus.cc:2454] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:13:59.545403 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19d80bad46460087c143778a4b0638" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:13:59.545506 7029 raft_consensus.cc:2454] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:13:59.545620 7028 raft_consensus.cc:3037] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:59.545579 6702 leader_election.cc:302] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:59.545892 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19d80bad46460087c143778a4b0638" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:13:59.545967 7842 raft_consensus.cc:2781] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:59.546080 6769 raft_consensus.cc:3037] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:59.546149 7842 raft_consensus.cc:686] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Becoming Leader. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:13:59.546720 7842 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.548311 7028 raft_consensus.cc:2454] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:13:59.548542 6769 raft_consensus.cc:2454] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:13:59.549171 6573 leader_election.cc:302] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:13:59.549517 7881 raft_consensus.cc:2781] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:59.549845 7881 raft_consensus.cc:686] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:13:59.550204 7881 consensus_queue.cc:237] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:13:59.550207 6490 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2). New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:13:59.553520 6493 catalog_manager.cc:5484] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:13:59.682380 7860 raft_consensus.cc:482] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:59.682621 7860 raft_consensus.cc:504] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:59.684407 7860 leader_election.cc:288] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:59.703814 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399d1a392b3f4d9e8531b6e0b5ad2a61" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:13:59.703874 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399d1a392b3f4d9e8531b6e0b5ad2a61" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:13:59.704090 6638 raft_consensus.cc:2454] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 0.
I20240501 08:13:59.704130 6769 raft_consensus.cc:2454] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 0.
I20240501 08:13:59.704655 6832 leader_election.cc:302] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:59.705061 7860 raft_consensus.cc:2781] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:59.705180 7860 raft_consensus.cc:482] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:59.705272 7860 raft_consensus.cc:3037] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:59.707391 7860 raft_consensus.cc:504] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:59.708086 7860 leader_election.cc:288] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:13:59.708670 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399d1a392b3f4d9e8531b6e0b5ad2a61" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:13:59.708629 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399d1a392b3f4d9e8531b6e0b5ad2a61" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:13:59.708881 6638 raft_consensus.cc:3037] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:59.708900 6769 raft_consensus.cc:3037] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:59.711767 6638 raft_consensus.cc:2454] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 1.
I20240501 08:13:59.711756 6769 raft_consensus.cc:2454] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 1.
I20240501 08:13:59.712358 6832 leader_election.cc:302] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:13:59.712733 7860 raft_consensus.cc:2781] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:59.713158 7860 raft_consensus.cc:686] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Becoming Leader. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Running, Role: LEADER
I20240501 08:13:59.713627 7860 consensus_queue.cc:237] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:13:59.717172 6493 catalog_manager.cc:5484] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: term changed from 0 to 1, leader changed from <none> to e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3). New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:14:00.114508 7842 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:00.121351 7842 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:00.132241 8005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:00.132584 8005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:14:00.136541 7881 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:00.141891 8017 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:00.142062 8005 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 08:14:00.202035 7860 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:00.210129 7860 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
INSERT report
rows total: 2000
time total: 213.357 ms
time per row: 0.106679 ms
I20240501 08:14:00.507573 6490 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42016:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:00.517628 7008 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:00.518169 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:00.518502 6879 tablet_service.cc:1467] Processing CreateTablet for tablet e9629fbbb4554edc9ba341d42b51f252 (DEFAULT_TABLE table=rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:00.518638 6878 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:00.518898 6879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9629fbbb4554edc9ba341d42b51f252. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:00.519115 6878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:00.519094 6616 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:00.519500 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:00.523341 6877 tablet_service.cc:1467] Processing CreateTablet for tablet 37a6157d7d734a80900139944481928c (DEFAULT_TABLE table=rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:00.523300 6747 tablet_service.cc:1467] Processing CreateTablet for tablet 37a6157d7d734a80900139944481928c (DEFAULT_TABLE table=rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:00.523391 7007 tablet_service.cc:1467] Processing CreateTablet for tablet e9629fbbb4554edc9ba341d42b51f252 (DEFAULT_TABLE table=rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:00.523738 6877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a6157d7d734a80900139944481928c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:00.523738 6747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a6157d7d734a80900139944481928c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:00.523790 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9629fbbb4554edc9ba341d42b51f252. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:00.527608 6618 tablet_service.cc:1467] Processing CreateTablet for tablet e9629fbbb4554edc9ba341d42b51f252 (DEFAULT_TABLE table=rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:00.527633 6617 tablet_service.cc:1467] Processing CreateTablet for tablet 37a6157d7d734a80900139944481928c (DEFAULT_TABLE table=rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:00.528174 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9629fbbb4554edc9ba341d42b51f252. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:00.530577 6617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a6157d7d734a80900139944481928c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:00.537959 8077 tablet_bootstrap.cc:492] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:00.540215 8076 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:00.540930 8075 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:00.542038 8076 tablet_bootstrap.cc:654] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:00.542330 8077 tablet_bootstrap.cc:654] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:00.544260 8075 tablet_bootstrap.cc:654] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:00.548271 8077 tablet_bootstrap.cc:492] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:14:00.548549 8077 ts_tablet_manager.cc:1367] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 08:14:00.549755 8077 raft_consensus.cc:348] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:00.550154 8077 raft_consensus.cc:374] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:00.550321 8077 raft_consensus.cc:729] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:00.550925 8077 consensus_queue.cc:260] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:00.551187 8078 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:00.551738 8077 ts_tablet_manager.cc:1398] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:00.553062 8075 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:00.553220 8075 ts_tablet_manager.cc:1367] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 08:14:00.554116 8075 raft_consensus.cc:348] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:00.554369 8075 raft_consensus.cc:374] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:00.554451 8075 raft_consensus.cc:729] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:00.554579 8078 tablet_bootstrap.cc:654] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:00.554672 8075 consensus_queue.cc:260] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:00.555193 8075 ts_tablet_manager.cc:1398] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:00.555470 8075 tablet_bootstrap.cc:492] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:00.555970 8076 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:00.556145 8076 ts_tablet_manager.cc:1367] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20240501 08:14:00.557052 8075 tablet_bootstrap.cc:654] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:00.557021 8076 raft_consensus.cc:348] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:00.557246 8076 raft_consensus.cc:374] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:00.557334 8076 raft_consensus.cc:729] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:00.557646 8076 consensus_queue.cc:260] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:00.558416 8076 ts_tablet_manager.cc:1398] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 08:14:00.558717 8076 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:00.560514 8076 tablet_bootstrap.cc:654] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:00.566430 8078 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:00.566644 8078 ts_tablet_manager.cc:1367] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20240501 08:14:00.567853 8078 raft_consensus.cc:348] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:00.568121 8078 raft_consensus.cc:374] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:00.568215 8078 raft_consensus.cc:729] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:00.568506 8078 consensus_queue.cc:260] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:00.572741 8078 ts_tablet_manager.cc:1398] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:14:00.573201 8076 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:00.573392 8076 ts_tablet_manager.cc:1367] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240501 08:14:00.574232 8076 raft_consensus.cc:348] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:00.574545 8076 raft_consensus.cc:374] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:00.574637 8076 raft_consensus.cc:729] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:00.574923 8076 consensus_queue.cc:260] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:00.575683 8076 ts_tablet_manager.cc:1398] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:14:00.576005 8076 tablet_bootstrap.cc:492] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:00.576545 8078 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:00.577283 8066 raft_consensus.cc:482] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:00.577459 8066 raft_consensus.cc:504] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:00.577751 8076 tablet_bootstrap.cc:654] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:00.578330 8066 leader_election.cc:288] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:00.579437 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a6157d7d734a80900139944481928c" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:14:00.580214 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a6157d7d734a80900139944481928c" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:14:00.583638 8078 tablet_bootstrap.cc:654] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:00.585192 8075 tablet_bootstrap.cc:492] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:00.585469 8075 ts_tablet_manager.cc:1367] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.018s
I20240501 08:14:00.586396 8075 raft_consensus.cc:348] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:00.586707 8075 raft_consensus.cc:374] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:00.586820 8075 raft_consensus.cc:729] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:00.587172 8075 consensus_queue.cc:260] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:00.587730 8076 tablet_bootstrap.cc:492] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:00.587881 8075 ts_tablet_manager.cc:1398] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:00.587925 8076 ts_tablet_manager.cc:1367] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240501 08:14:00.588318 8075 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:00.588897 7972 raft_consensus.cc:482] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:00.588953 8076 raft_consensus.cc:348] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:00.589171 8076 raft_consensus.cc:374] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:00.589069 7972 raft_consensus.cc:504] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:00.589262 8076 raft_consensus.cc:729] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:00.589533 8076 consensus_queue.cc:260] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:00.589845 7972 leader_election.cc:288] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:00.590391 8075 tablet_bootstrap.cc:654] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:00.590720 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9629fbbb4554edc9ba341d42b51f252" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:14:00.590945 8076 ts_tablet_manager.cc:1398] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:14:00.591346 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9629fbbb4554edc9ba341d42b51f252" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
W20240501 08:14:00.592411 6962 leader_election.cc:341] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:14:00.592689 8075 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:00.590941 6899 raft_consensus.cc:2454] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:14:00.592940 8075 ts_tablet_manager.cc:1367] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:14:00.593183 6962 leader_election.cc:302] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, e3a1c4a5b44340be8a183d5d649b27ac; no voters: 97cdee0bf9d84645a1451f9822c5d699
I20240501 08:14:00.593833 7972 raft_consensus.cc:2781] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:00.593959 7972 raft_consensus.cc:482] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:00.594059 7972 raft_consensus.cc:3037] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:00.593804 8075 raft_consensus.cc:348] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:00.594230 8075 raft_consensus.cc:374] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:00.594393 8075 raft_consensus.cc:729] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:00.594700 8075 consensus_queue.cc:260] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:00.595326 8075 ts_tablet_manager.cc:1398] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:00.596678 7972 raft_consensus.cc:504] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:00.597292 7972 leader_election.cc:288] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Requested vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:00.597972 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9629fbbb4554edc9ba341d42b51f252" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:00.597999 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9629fbbb4554edc9ba341d42b51f252" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:00.598186 6899 raft_consensus.cc:3037] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:00.598205 6638 raft_consensus.cc:3037] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:00.601034 6638 raft_consensus.cc:2454] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:14:00.601045 6899 raft_consensus.cc:2454] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:14:00.601682 6962 leader_election.cc:302] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:00.603049 7972 raft_consensus.cc:2781] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:00.603556 7972 raft_consensus.cc:686] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:14:00.603920 7972 consensus_queue.cc:237] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
W20240501 08:14:00.607067 6702 leader_election.cc:341] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): Illegal state: must be running to vote when last-logged opid is not known
W20240501 08:14:00.607439 6702 leader_election.cc:341] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:14:00.607575 6702 leader_election.cc:302] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 97cdee0bf9d84645a1451f9822c5d699, e3a1c4a5b44340be8a183d5d649b27ac
I20240501 08:14:00.607937 8066 raft_consensus.cc:2726] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:14:00.608314 6492 catalog_manager.cc:5484] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:14:00.609406 8078 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:00.609622 8078 ts_tablet_manager.cc:1367] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.007s
I20240501 08:14:00.610497 8078 raft_consensus.cc:348] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:00.610808 8078 raft_consensus.cc:374] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:00.610929 8078 raft_consensus.cc:729] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:00.611191 8078 consensus_queue.cc:260] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:00.611776 8078 ts_tablet_manager.cc:1398] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:00.637533 8017 raft_consensus.cc:482] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:00.637768 8017 raft_consensus.cc:504] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:00.638592 8017 leader_election.cc:288] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:00.639161 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9bd7500e8747c6b8ee94a1faa1c5f7" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:14:00.639137 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9bd7500e8747c6b8ee94a1faa1c5f7" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:14:00.639396 7028 raft_consensus.cc:2454] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:00.639425 6899 raft_consensus.cc:2454] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:00.639889 6573 leader_election.cc:302] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:14:00.640216 8017 raft_consensus.cc:2781] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:00.640327 8017 raft_consensus.cc:482] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:00.640430 8017 raft_consensus.cc:3037] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:00.642410 8017 raft_consensus.cc:504] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:00.643088 8017 leader_election.cc:288] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:00.643672 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9bd7500e8747c6b8ee94a1faa1c5f7" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:00.643661 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9bd7500e8747c6b8ee94a1faa1c5f7" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:00.643883 7028 raft_consensus.cc:3037] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:00.643903 6899 raft_consensus.cc:3037] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:00.646762 6899 raft_consensus.cc:2454] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:00.646777 7028 raft_consensus.cc:2454] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:00.647377 6573 leader_election.cc:302] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:00.647686 8017 raft_consensus.cc:2781] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:00.647837 8017 raft_consensus.cc:686] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:00.648090 8017 consensus_queue.cc:237] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:00.651062 6492 catalog_manager.cc:5484] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:00.683040 8017 raft_consensus.cc:482] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:00.683241 8017 raft_consensus.cc:504] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:00.683934 8017 leader_election.cc:288] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:00.684465 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a6157d7d734a80900139944481928c" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:14:00.684479 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a6157d7d734a80900139944481928c" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:14:00.684690 6769 raft_consensus.cc:2454] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:00.684703 6899 raft_consensus.cc:2454] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:00.685158 6571 leader_election.cc:302] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:00.685518 8017 raft_consensus.cc:2781] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:00.685645 8017 raft_consensus.cc:482] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:00.685737 8017 raft_consensus.cc:3037] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:00.708256 8017 raft_consensus.cc:504] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:00.708968 8017 leader_election.cc:288] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:00.709635 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a6157d7d734a80900139944481928c" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:00.709643 6899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a6157d7d734a80900139944481928c" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:00.709867 6769 raft_consensus.cc:3037] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:00.709884 6899 raft_consensus.cc:3037] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:00.711944 6899 raft_consensus.cc:2454] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:00.711952 6769 raft_consensus.cc:2454] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:00.712445 6571 leader_election.cc:302] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:00.712797 8017 raft_consensus.cc:2781] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:00.712935 8017 raft_consensus.cc:686] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:00.713213 8017 consensus_queue.cc:237] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:00.716181 6492 catalog_manager.cc:5484] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:14:01.096414 7972 consensus_queue.cc:1035] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:01.102078 8088 consensus_queue.cc:1035] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:01.104223 8092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:01.104676 8092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:01.114995 8092 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 08:14:01.212978 8017 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:01.225744 8122 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:01.274369 8122 consensus_queue.cc:1035] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240501 08:14:01.292328 8122 consensus_queue.cc:1035] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:01.396229 8134 mvcc.cc:205] Tried to move back new op lower bound from 7022801884408897536 to 7022801881963393024. Current Snapshot: MvccSnapshot[applied={T|T < 7022801884408897536}]
INSERT report
rows total: 2000
time total: 251.056 ms
time per row: 0.125528 ms
I20240501 08:14:01.515755 6490 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42016:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:01.525959 7007 tablet_service.cc:1467] Processing CreateTablet for tablet 491674dfbf74498e8a787bebaa5e95da (DEFAULT_TABLE table=rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:01.526037 6747 tablet_service.cc:1467] Processing CreateTablet for tablet 491674dfbf74498e8a787bebaa5e95da (DEFAULT_TABLE table=rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:01.526407 6749 tablet_service.cc:1467] Processing CreateTablet for tablet 6575b94bd51546f1bd3bb862f012e7ca (DEFAULT_TABLE table=rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:01.526512 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491674dfbf74498e8a787bebaa5e95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:01.526641 7008 tablet_service.cc:1467] Processing CreateTablet for tablet 6575b94bd51546f1bd3bb862f012e7ca (DEFAULT_TABLE table=rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:01.526827 6616 tablet_service.cc:1467] Processing CreateTablet for tablet 491674dfbf74498e8a787bebaa5e95da (DEFAULT_TABLE table=rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:01.526996 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6575b94bd51546f1bd3bb862f012e7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:01.527236 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491674dfbf74498e8a787bebaa5e95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:01.528394 6878 tablet_service.cc:1467] Processing CreateTablet for tablet 6575b94bd51546f1bd3bb862f012e7ca (DEFAULT_TABLE table=rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:01.528805 6878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6575b94bd51546f1bd3bb862f012e7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:01.529971 6748 tablet_service.cc:1467] Processing CreateTablet for tablet 45d076303ea24d7fa0fed5a9d132019c (DEFAULT_TABLE table=rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:01.530167 6747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491674dfbf74498e8a787bebaa5e95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:01.530968 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d076303ea24d7fa0fed5a9d132019c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:01.534564 6749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6575b94bd51546f1bd3bb862f012e7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:01.534816 7009 tablet_service.cc:1467] Processing CreateTablet for tablet 45d076303ea24d7fa0fed5a9d132019c (DEFAULT_TABLE table=rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:01.535331 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d076303ea24d7fa0fed5a9d132019c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:01.537492 8155 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:01.539705 6616 tablet_service.cc:1467] Processing CreateTablet for tablet 45d076303ea24d7fa0fed5a9d132019c (DEFAULT_TABLE table=rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:01.540143 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d076303ea24d7fa0fed5a9d132019c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:01.547880 8155 tablet_bootstrap.cc:654] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:01.548640 8157 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:01.551072 8157 tablet_bootstrap.cc:654] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:01.554515 8156 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:01.556561 8156 tablet_bootstrap.cc:654] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:01.562680 8156 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:01.562935 8156 ts_tablet_manager.cc:1367] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240501 08:14:01.563745 8156 raft_consensus.cc:348] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:01.563957 8156 raft_consensus.cc:374] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:01.564057 8156 raft_consensus.cc:729] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:01.564271 8156 consensus_queue.cc:260] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:01.564863 8156 ts_tablet_manager.cc:1398] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:14:01.567315 8156 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:01.569113 8156 tablet_bootstrap.cc:654] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:01.570521 8161 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:01.571887 8161 tablet_bootstrap.cc:654] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:01.573858 8157 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:01.573937 8161 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:14:01.574087 8161 ts_tablet_manager.cc:1367] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:14:01.574121 8157 ts_tablet_manager.cc:1367] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.004s
I20240501 08:14:01.575002 8157 raft_consensus.cc:348] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:01.574995 8161 raft_consensus.cc:348] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:01.575227 8161 raft_consensus.cc:374] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:01.575314 8161 raft_consensus.cc:729] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:01.575364 8157 raft_consensus.cc:374] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:01.575567 8157 raft_consensus.cc:729] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:01.575601 8161 consensus_queue.cc:260] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:01.576340 8161 ts_tablet_manager.cc:1398] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:14:01.576648 8161 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:01.576604 8155 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:01.576855 8155 ts_tablet_manager.cc:1367] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.000s
I20240501 08:14:01.577593 8157 consensus_queue.cc:260] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:01.577803 8155 raft_consensus.cc:348] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:01.578080 8155 raft_consensus.cc:374] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:01.578176 8155 raft_consensus.cc:729] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:01.578619 8157 ts_tablet_manager.cc:1398] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:14:01.578595 8155 consensus_queue.cc:260] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:01.579396 8155 ts_tablet_manager.cc:1398] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:01.579676 8155 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:01.581670 8161 tablet_bootstrap.cc:654] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:01.583300 8155 tablet_bootstrap.cc:654] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:01.585960 8156 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:01.586277 8156 ts_tablet_manager.cc:1367] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20240501 08:14:01.587333 8156 raft_consensus.cc:348] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:01.587740 8156 raft_consensus.cc:374] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:01.587838 8156 raft_consensus.cc:729] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:01.588186 8156 consensus_queue.cc:260] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:01.588991 8156 ts_tablet_manager.cc:1398] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:14:01.589068 8155 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:01.589247 8155 ts_tablet_manager.cc:1367] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 08:14:01.589454 8156 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:01.590077 8155 raft_consensus.cc:348] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:01.590348 8155 raft_consensus.cc:374] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:01.590431 8155 raft_consensus.cc:729] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:01.590724 8155 consensus_queue.cc:260] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:01.591584 8156 tablet_bootstrap.cc:654] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:01.592531 8161 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:14:01.592703 8161 ts_tablet_manager.cc:1367] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.007s
I20240501 08:14:01.593461 8161 raft_consensus.cc:348] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:01.593693 8161 raft_consensus.cc:374] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:01.593756 8161 raft_consensus.cc:729] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:01.594028 8161 consensus_queue.cc:260] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:01.594192 8155 ts_tablet_manager.cc:1398] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:14:01.594729 8161 ts_tablet_manager.cc:1398] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:01.595465 8161 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:01.597272 8161 tablet_bootstrap.cc:654] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:01.598140 8156 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:01.598562 8156 ts_tablet_manager.cc:1367] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240501 08:14:01.599583 8156 raft_consensus.cc:348] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:01.599857 8156 raft_consensus.cc:374] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:01.599968 8156 raft_consensus.cc:729] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:01.600423 8156 consensus_queue.cc:260] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:01.601050 8156 ts_tablet_manager.cc:1398] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:14:01.602217 8161 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:14:01.602408 8161 ts_tablet_manager.cc:1367] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240501 08:14:01.603085 8161 raft_consensus.cc:348] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:01.603287 8161 raft_consensus.cc:374] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:01.603387 8161 raft_consensus.cc:729] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:01.603662 8161 consensus_queue.cc:260] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:01.604305 8161 ts_tablet_manager.cc:1398] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:01.615311 8122 raft_consensus.cc:482] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:01.615438 8135 raft_consensus.cc:482] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:01.615556 8122 raft_consensus.cc:504] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:01.615592 8135 raft_consensus.cc:504] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:01.616357 8122 leader_election.cc:288] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:01.616382 8135 leader_election.cc:288] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:01.616993 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d076303ea24d7fa0fed5a9d132019c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:14:01.617033 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491674dfbf74498e8a787bebaa5e95da" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:14:01.616999 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d076303ea24d7fa0fed5a9d132019c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:14:01.616997 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491674dfbf74498e8a787bebaa5e95da" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:14:01.617218 6637 raft_consensus.cc:2454] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:14:01.617256 6768 raft_consensus.cc:2454] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:14:01.617250 7028 raft_consensus.cc:2454] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:01.617298 6769 raft_consensus.cc:2454] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:01.617691 6962 leader_election.cc:302] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:14:01.617720 6573 leader_election.cc:302] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:14:01.617995 8135 raft_consensus.cc:2781] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:01.618079 8122 raft_consensus.cc:2781] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:01.618120 8135 raft_consensus.cc:482] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:01.618172 8122 raft_consensus.cc:482] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:01.618198 8135 raft_consensus.cc:3037] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:01.618299 8122 raft_consensus.cc:3037] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:01.620266 8122 raft_consensus.cc:504] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:01.620285 8135 raft_consensus.cc:504] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:01.620988 8135 leader_election.cc:288] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:01.621344 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491674dfbf74498e8a787bebaa5e95da" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:01.621366 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d076303ea24d7fa0fed5a9d132019c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:01.621558 6637 raft_consensus.cc:3037] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:01.621585 7028 raft_consensus.cc:3037] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:01.621505 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491674dfbf74498e8a787bebaa5e95da" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:01.621502 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d076303ea24d7fa0fed5a9d132019c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:01.621757 6768 raft_consensus.cc:3037] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:01.621784 6769 raft_consensus.cc:3037] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:01.624435 6637 raft_consensus.cc:2454] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:14:01.624423 7028 raft_consensus.cc:2454] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:01.624615 6769 raft_consensus.cc:2454] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:14:01.624929 6573 leader_election.cc:302] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:14:01.624943 6962 leader_election.cc:302] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:14:01.625257 8017 raft_consensus.cc:2781] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:01.625312 8135 raft_consensus.cc:2781] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:01.625384 8017 raft_consensus.cc:686] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:01.625458 8135 raft_consensus.cc:686] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:14:01.625782 8017 consensus_queue.cc:237] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:01.625943 8135 consensus_queue.cc:237] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:01.624716 6768 raft_consensus.cc:2454] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:01.626734 8122 leader_election.cc:288] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:01.636111 6492 catalog_manager.cc:5484] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:14:01.637667 6490 catalog_manager.cc:5484] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:14:01.899677 7860 raft_consensus.cc:482] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:01.899896 7860 raft_consensus.cc:504] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:01.900604 7860 leader_election.cc:288] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:01.901294 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6575b94bd51546f1bd3bb862f012e7ca" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:14:01.901567 6768 raft_consensus.cc:2454] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 0.
I20240501 08:14:01.902041 6835 leader_election.cc:302] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:01.902431 7860 raft_consensus.cc:2781] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:01.902578 7860 raft_consensus.cc:482] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:01.902716 7860 raft_consensus.cc:3037] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:01.905292 7860 raft_consensus.cc:504] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:01.905995 7860 leader_election.cc:288] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:01.906590 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6575b94bd51546f1bd3bb862f012e7ca" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:01.906801 6768 raft_consensus.cc:3037] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:01.907920 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6575b94bd51546f1bd3bb862f012e7ca" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:14:01.907982 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6575b94bd51546f1bd3bb862f012e7ca" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:01.908169 7028 raft_consensus.cc:2454] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 0.
I20240501 08:14:01.909277 6768 raft_consensus.cc:2454] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 1.
I20240501 08:14:01.909613 6835 leader_election.cc:302] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters: 28b7613bd4834f3ab245c82dd7e80240
I20240501 08:14:01.909873 7860 raft_consensus.cc:2781] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:01.910225 7860 raft_consensus.cc:686] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Becoming Leader. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Running, Role: LEADER
I20240501 08:14:01.910497 7860 consensus_queue.cc:237] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:01.913247 6490 catalog_manager.cc:5484] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: term changed from 0 to 1, leader changed from <none> to e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3). New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:14:02.024763 8122 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:02.047097 8122 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:02.096102 8135 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:02.101536 7972 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:02.313819 8174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:02.314199 8174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:02.322137 8174 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 08:14:02.405918 7860 consensus_queue.cc:1035] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:02.450235 7029 raft_consensus.cc:3037] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:02.454100 7860 consensus_queue.cc:1035] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 282.207 ms
time per row: 0.141104 ms
I20240501 08:14:02.753538 6492 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42016:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:02.763842 6616 tablet_service.cc:1467] Processing CreateTablet for tablet 658f2a172a2d47c8bf8b44c9d51132e4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:02.764181 6748 tablet_service.cc:1467] Processing CreateTablet for tablet 658f2a172a2d47c8bf8b44c9d51132e4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:02.764382 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658f2a172a2d47c8bf8b44c9d51132e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.764534 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658f2a172a2d47c8bf8b44c9d51132e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.765695 7008 tablet_service.cc:1467] Processing CreateTablet for tablet f93c2d0f9c7947bb83a50c7fb8c3518a (DEFAULT_TABLE table=rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:02.766186 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93c2d0f9c7947bb83a50c7fb8c3518a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.767573 6877 tablet_service.cc:1467] Processing CreateTablet for tablet 60238162349647c681685349c6f8ede4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:02.768007 6877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60238162349647c681685349c6f8ede4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.770709 7007 tablet_service.cc:1467] Processing CreateTablet for tablet 60238162349647c681685349c6f8ede4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:02.770769 6618 tablet_service.cc:1467] Processing CreateTablet for tablet 60238162349647c681685349c6f8ede4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:02.771112 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60238162349647c681685349c6f8ede4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.771188 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60238162349647c681685349c6f8ede4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.771783 6747 tablet_service.cc:1467] Processing CreateTablet for tablet f93c2d0f9c7947bb83a50c7fb8c3518a (DEFAULT_TABLE table=rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:02.772167 6747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93c2d0f9c7947bb83a50c7fb8c3518a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.774520 6878 tablet_service.cc:1467] Processing CreateTablet for tablet 658f2a172a2d47c8bf8b44c9d51132e4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:02.774724 6879 tablet_service.cc:1467] Processing CreateTablet for tablet f93c2d0f9c7947bb83a50c7fb8c3518a (DEFAULT_TABLE table=rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:02.775049 6878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658f2a172a2d47c8bf8b44c9d51132e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.775246 6879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93c2d0f9c7947bb83a50c7fb8c3518a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.785727 8242 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:02.788211 8242 tablet_bootstrap.cc:654] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:02.788673 8241 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:02.789827 8240 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:02.791052 8241 tablet_bootstrap.cc:654] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:02.791443 8239 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:02.792061 8240 tablet_bootstrap.cc:654] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:02.793205 8239 tablet_bootstrap.cc:654] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:02.796672 8239 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:14:02.796727 8240 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:02.796844 8239 ts_tablet_manager.cc:1367] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 08:14:02.796870 8240 ts_tablet_manager.cc:1367] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 08:14:02.797427 8241 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:02.797582 8241 ts_tablet_manager.cc:1367] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 08:14:02.797627 8240 raft_consensus.cc:348] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:02.797644 8239 raft_consensus.cc:348] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:02.797842 8239 raft_consensus.cc:374] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:02.797963 8239 raft_consensus.cc:729] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.798097 8240 raft_consensus.cc:374] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:02.798216 8240 raft_consensus.cc:729] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.798211 8239 consensus_queue.cc:260] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:02.798419 8241 raft_consensus.cc:348] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.798619 8241 raft_consensus.cc:374] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:02.798713 8241 raft_consensus.cc:729] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.798740 8240 consensus_queue.cc:260] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:02.798967 8241 consensus_queue.cc:260] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.799459 8242 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:02.799643 8242 ts_tablet_manager.cc:1367] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 08:14:02.800029 8241 ts_tablet_manager.cc:1398] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:02.800335 8241 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:02.802349 8241 tablet_bootstrap.cc:654] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:02.802816 8239 ts_tablet_manager.cc:1398] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 08:14:02.803529 8242 raft_consensus.cc:348] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.803771 8239 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:02.803784 8242 raft_consensus.cc:374] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:02.803902 8242 raft_consensus.cc:729] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.804140 8242 consensus_queue.cc:260] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.804869 8242 ts_tablet_manager.cc:1398] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:14:02.805193 8242 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:02.806176 8239 tablet_bootstrap.cc:654] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:02.806764 8242 tablet_bootstrap.cc:654] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:02.810775 8239 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:14:02.810972 8239 ts_tablet_manager.cc:1367] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:14:02.811676 8240 ts_tablet_manager.cc:1398] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 08:14:02.811833 8239 raft_consensus.cc:348] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.812103 8239 raft_consensus.cc:374] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:02.812177 8239 raft_consensus.cc:729] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.812436 8239 consensus_queue.cc:260] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.813871 8239 ts_tablet_manager.cc:1398] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:02.814752 8240 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:02.816781 8242 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:02.817026 8242 ts_tablet_manager.cc:1367] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20240501 08:14:02.817701 8242 raft_consensus.cc:348] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.818109 8242 raft_consensus.cc:374] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:02.818192 8242 raft_consensus.cc:729] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.818440 8240 tablet_bootstrap.cc:654] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:02.818470 8242 consensus_queue.cc:260] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.819011 8242 ts_tablet_manager.cc:1398] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:14:02.820537 8240 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:02.820739 8240 ts_tablet_manager.cc:1367] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240501 08:14:02.821616 8240 raft_consensus.cc:348] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.821966 8240 raft_consensus.cc:374] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:02.822060 8240 raft_consensus.cc:729] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.822388 8240 consensus_queue.cc:260] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.823096 8240 ts_tablet_manager.cc:1398] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 08:14:02.824260 8241 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:02.824431 8241 ts_tablet_manager.cc:1367] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.013s
I20240501 08:14:02.825217 8241 raft_consensus.cc:348] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.825438 8241 raft_consensus.cc:374] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:02.825515 8241 raft_consensus.cc:729] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.825798 8241 consensus_queue.cc:260] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.826366 8241 ts_tablet_manager.cc:1398] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:14:02.826758 8241 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:02.828305 8241 tablet_bootstrap.cc:654] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:02.830374 8241 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:02.830531 8241 ts_tablet_manager.cc:1367] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:14:02.831231 8241 raft_consensus.cc:348] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:02.831419 8241 raft_consensus.cc:374] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:02.831509 8241 raft_consensus.cc:729] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.831719 8241 consensus_queue.cc:260] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:02.832190 8241 ts_tablet_manager.cc:1398] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:14:02.832890 8206 raft_consensus.cc:482] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:02.833037 8206 raft_consensus.cc:504] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:02.833806 8206 leader_election.cc:288] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:02.834400 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658f2a172a2d47c8bf8b44c9d51132e4" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:14:02.834617 6768 raft_consensus.cc:2454] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 0.
I20240501 08:14:02.834633 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658f2a172a2d47c8bf8b44c9d51132e4" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:14:02.834878 6637 raft_consensus.cc:2454] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 0.
I20240501 08:14:02.835068 6835 leader_election.cc:302] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:02.835350 8206 raft_consensus.cc:2781] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:02.835465 8206 raft_consensus.cc:482] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:02.835569 8206 raft_consensus.cc:3037] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:02.838094 8206 raft_consensus.cc:504] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:02.838709 8206 leader_election.cc:288] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 election: Requested vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:02.839273 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658f2a172a2d47c8bf8b44c9d51132e4" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:02.839264 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658f2a172a2d47c8bf8b44c9d51132e4" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:02.839474 6768 raft_consensus.cc:3037] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:02.839493 6637 raft_consensus.cc:3037] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:02.842326 6768 raft_consensus.cc:2454] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 1.
I20240501 08:14:02.842314 6637 raft_consensus.cc:2454] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 1.
I20240501 08:14:02.842792 6835 leader_election.cc:302] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:02.843152 8206 raft_consensus.cc:2781] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:02.843358 8206 raft_consensus.cc:686] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Becoming Leader. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Running, Role: LEADER
I20240501 08:14:02.843629 8206 consensus_queue.cc:237] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:02.847662 6494 catalog_manager.cc:5484] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: term changed from 0 to 1, leader changed from <none> to e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3). New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:14:02.863343 8066 raft_consensus.cc:482] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:02.863533 8066 raft_consensus.cc:504] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.864174 8066 leader_election.cc:288] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:02.864713 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93c2d0f9c7947bb83a50c7fb8c3518a" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:14:02.864748 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93c2d0f9c7947bb83a50c7fb8c3518a" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:14:02.864957 6898 raft_consensus.cc:2454] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:14:02.864982 7029 raft_consensus.cc:2454] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:14:02.865420 6704 leader_election.cc:302] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:14:02.865701 8066 raft_consensus.cc:2781] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:02.865834 8066 raft_consensus.cc:482] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:02.865947 8066 raft_consensus.cc:3037] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:02.868093 8066 raft_consensus.cc:504] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.868623 8066 leader_election.cc:288] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Requested vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:02.869072 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93c2d0f9c7947bb83a50c7fb8c3518a" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:02.869064 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f93c2d0f9c7947bb83a50c7fb8c3518a" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:02.869251 6898 raft_consensus.cc:3037] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:02.869263 7029 raft_consensus.cc:3037] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:02.871109 7029 raft_consensus.cc:2454] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:14:02.871140 6898 raft_consensus.cc:2454] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:14:02.871519 6704 leader_election.cc:302] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:02.871790 8066 raft_consensus.cc:2781] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:02.872174 8066 raft_consensus.cc:686] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Becoming Leader. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:14:02.872467 8066 consensus_queue.cc:237] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.875497 6494 catalog_manager.cc:5484] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2). New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:14:02.888594 8181 raft_consensus.cc:482] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:02.888768 8181 raft_consensus.cc:504] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.889377 8181 leader_election.cc:288] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:02.889816 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60238162349647c681685349c6f8ede4" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:14:02.889909 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60238162349647c681685349c6f8ede4" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:14:02.890029 6637 raft_consensus.cc:2454] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:14:02.890157 6898 raft_consensus.cc:2454] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 0.
I20240501 08:14:02.890431 6962 leader_election.cc:302] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:14:02.890650 8181 raft_consensus.cc:2781] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:02.890753 8181 raft_consensus.cc:482] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:02.890837 8181 raft_consensus.cc:3037] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:02.892742 8181 raft_consensus.cc:504] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.893265 8181 leader_election.cc:288] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Requested vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:02.893714 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60238162349647c681685349c6f8ede4" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:02.893751 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60238162349647c681685349c6f8ede4" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:02.893889 6637 raft_consensus.cc:3037] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:02.893930 6898 raft_consensus.cc:3037] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:02.896111 6898 raft_consensus.cc:2454] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:14:02.896230 6637 raft_consensus.cc:2454] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 1.
I20240501 08:14:02.896576 6962 leader_election.cc:302] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:02.896864 8181 raft_consensus.cc:2781] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:02.897048 8181 raft_consensus.cc:686] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:14:02.897332 8181 consensus_queue.cc:237] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:02.900900 6494 catalog_manager.cc:5484] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:14:03.259610 8206 consensus_queue.cc:1035] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:03.268546 8206 consensus_queue.cc:1035] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:03.288261 8256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:03.288630 8256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:03.296202 8256 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 08:14:03.340909 8066 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:03.345547 7972 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:03.351204 8253 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:03.353075 7972 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 342.681 ms
time per row: 0.17134 ms
I20240501 08:14:03.793187 6494 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42016:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:03.802495 6748 tablet_service.cc:1467] Processing CreateTablet for tablet 13b4e2d5fe4846e8b3c6081abb3c53da (DEFAULT_TABLE table=rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:03.803012 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b4e2d5fe4846e8b3c6081abb3c53da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:03.802944 6747 tablet_service.cc:1467] Processing CreateTablet for tablet a25e1ddb987b4f8c8a4031053f581c7c (DEFAULT_TABLE table=rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:03.802958 6618 tablet_service.cc:1467] Processing CreateTablet for tablet 13b4e2d5fe4846e8b3c6081abb3c53da (DEFAULT_TABLE table=rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:03.803282 6747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25e1ddb987b4f8c8a4031053f581c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:03.803355 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b4e2d5fe4846e8b3c6081abb3c53da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:03.803805 7008 tablet_service.cc:1467] Processing CreateTablet for tablet a25e1ddb987b4f8c8a4031053f581c7c (DEFAULT_TABLE table=rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:03.804229 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25e1ddb987b4f8c8a4031053f581c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:03.809974 6878 tablet_service.cc:1467] Processing CreateTablet for tablet 13b4e2d5fe4846e8b3c6081abb3c53da (DEFAULT_TABLE table=rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:03.810520 6878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b4e2d5fe4846e8b3c6081abb3c53da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:03.811331 6877 tablet_service.cc:1467] Processing CreateTablet for tablet d7903106d7e74e1ea102b1abdbfce63e (DEFAULT_TABLE table=rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:03.811735 6877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7903106d7e74e1ea102b1abdbfce63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:03.815201 7007 tablet_service.cc:1467] Processing CreateTablet for tablet d7903106d7e74e1ea102b1abdbfce63e (DEFAULT_TABLE table=rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:03.815371 6616 tablet_service.cc:1467] Processing CreateTablet for tablet a25e1ddb987b4f8c8a4031053f581c7c (DEFAULT_TABLE table=rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:03.815696 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7903106d7e74e1ea102b1abdbfce63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:03.815784 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25e1ddb987b4f8c8a4031053f581c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:03.824672 6749 tablet_service.cc:1467] Processing CreateTablet for tablet d7903106d7e74e1ea102b1abdbfce63e (DEFAULT_TABLE table=rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:03.825278 6749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7903106d7e74e1ea102b1abdbfce63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:03.828262 8316 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:03.828861 8317 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:03.830586 8316 tablet_bootstrap.cc:654] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:03.832773 8317 tablet_bootstrap.cc:654] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:03.834570 8320 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:03.836938 8320 tablet_bootstrap.cc:654] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:03.837734 8318 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:03.837920 8317 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:14:03.838536 8317 ts_tablet_manager.cc:1367] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240501 08:14:03.839582 8317 raft_consensus.cc:348] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:03.840003 8317 raft_consensus.cc:374] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:03.840154 8317 raft_consensus.cc:729] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:03.840530 8317 consensus_queue.cc:260] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:03.841465 8317 ts_tablet_manager.cc:1398] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:14:03.842175 8318 tablet_bootstrap.cc:654] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:03.842180 8317 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:03.844101 8317 tablet_bootstrap.cc:654] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:03.846750 8320 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:03.847023 8320 ts_tablet_manager.cc:1367] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240501 08:14:03.848044 8320 raft_consensus.cc:348] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:03.848380 8320 raft_consensus.cc:374] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:03.848539 8320 raft_consensus.cc:729] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:03.848991 8316 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:03.848896 8320 consensus_queue.cc:260] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:03.849180 8316 ts_tablet_manager.cc:1367] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20240501 08:14:03.850006 8316 raft_consensus.cc:348] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:03.850152 8317 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:14:03.850162 8318 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:03.850214 8316 raft_consensus.cc:374] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:03.850330 8316 raft_consensus.cc:729] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:03.850350 8317 ts_tablet_manager.cc:1367] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240501 08:14:03.850353 8318 ts_tablet_manager.cc:1367] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240501 08:14:03.850610 8316 consensus_queue.cc:260] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:03.851161 8318 raft_consensus.cc:348] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:03.851239 8317 raft_consensus.cc:348] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:03.851363 8318 raft_consensus.cc:374] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:03.851449 8318 raft_consensus.cc:729] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:03.851454 8317 raft_consensus.cc:374] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:03.851526 8317 raft_consensus.cc:729] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:03.851737 8318 consensus_queue.cc:260] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:03.851800 8317 consensus_queue.cc:260] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:03.852386 8317 ts_tablet_manager.cc:1398] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:03.852581 8318 ts_tablet_manager.cc:1398] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:03.852882 8317 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:03.852905 8318 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:03.854571 8317 tablet_bootstrap.cc:654] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:03.854573 8318 tablet_bootstrap.cc:654] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:03.856611 8320 ts_tablet_manager.cc:1398] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:14:03.862591 8316 ts_tablet_manager.cc:1398] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 08:14:03.864105 8320 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:03.866241 8316 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:03.866272 8320 tablet_bootstrap.cc:654] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:03.868261 8316 tablet_bootstrap.cc:654] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:03.872040 8316 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:03.872133 8318 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:03.872225 8316 ts_tablet_manager.cc:1367] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:14:03.872313 8318 ts_tablet_manager.cc:1367] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.007s
I20240501 08:14:03.872509 8320 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:03.872725 8320 ts_tablet_manager.cc:1367] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 08:14:03.873164 8317 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:14:03.873098 8316 raft_consensus.cc:348] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:03.873329 8316 raft_consensus.cc:374] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:03.873237 8318 raft_consensus.cc:348] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:03.873396 8316 raft_consensus.cc:729] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:03.873468 8318 raft_consensus.cc:374] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:03.873558 8318 raft_consensus.cc:729] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:03.873610 8317 ts_tablet_manager.cc:1367] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.011s
I20240501 08:14:03.873654 8316 consensus_queue.cc:260] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:03.873651 8320 raft_consensus.cc:348] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:03.873837 8318 consensus_queue.cc:260] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:03.874222 8316 ts_tablet_manager.cc:1398] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:14:03.874387 8318 ts_tablet_manager.cc:1398] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:14:03.873863 8320 raft_consensus.cc:374] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:03.875222 8317 raft_consensus.cc:348] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:03.875427 8317 raft_consensus.cc:374] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:03.875535 8317 raft_consensus.cc:729] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:03.875751 8317 consensus_queue.cc:260] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:03.876230 8317 ts_tablet_manager.cc:1398] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:14:03.874884 8320 raft_consensus.cc:729] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:03.877369 8320 consensus_queue.cc:260] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:03.878007 8320 ts_tablet_manager.cc:1398] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:14:03.936240 8066 raft_consensus.cc:482] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:03.936451 8066 raft_consensus.cc:504] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:03.937093 8066 leader_election.cc:288] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:03.937562 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:14:03.937570 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:14:03.937762 6638 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:14:03.937788 7029 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:14:03.938282 6704 leader_election.cc:302] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:14:03.938560 8066 raft_consensus.cc:2781] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:03.938673 8066 raft_consensus.cc:482] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:03.938768 8066 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:03.940671 8066 raft_consensus.cc:504] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:03.941267 8066 leader_election.cc:288] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:03.941818 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:03.941828 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:03.942006 6638 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:03.942011 7029 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:03.944393 6638 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:14:03.944408 7029 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:14:03.944854 6704 leader_election.cc:302] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:14:03.945179 8066 raft_consensus.cc:2781] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:03.945336 8066 raft_consensus.cc:686] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Becoming Leader. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:14:03.945612 8066 consensus_queue.cc:237] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:03.948760 6489 catalog_manager.cc:5484] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2). New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:14:03.968539 8066 raft_consensus.cc:482] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:03.968796 8066 raft_consensus.cc:504] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:03.969439 8066 leader_election.cc:288] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:03.970010 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7903106d7e74e1ea102b1abdbfce63e" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:14:03.969966 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7903106d7e74e1ea102b1abdbfce63e" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:14:03.970242 6898 raft_consensus.cc:2454] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:14:03.970284 7029 raft_consensus.cc:2454] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 0.
I20240501 08:14:03.970714 6702 leader_election.cc:302] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:03.971086 8066 raft_consensus.cc:2781] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:03.971172 8066 raft_consensus.cc:482] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:03.971264 8066 raft_consensus.cc:3037] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:03.973464 8066 raft_consensus.cc:504] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:03.974207 8066 leader_election.cc:288] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Requested vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:03.974721 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7903106d7e74e1ea102b1abdbfce63e" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:03.974745 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7903106d7e74e1ea102b1abdbfce63e" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:03.974911 6898 raft_consensus.cc:3037] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:03.974956 7029 raft_consensus.cc:3037] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:03.976922 7029 raft_consensus.cc:2454] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:14:03.976933 6898 raft_consensus.cc:2454] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 1.
I20240501 08:14:03.977432 6704 leader_election.cc:302] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:03.977880 8066 raft_consensus.cc:2781] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:03.978067 8066 raft_consensus.cc:686] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Becoming Leader. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:14:03.978385 8066 consensus_queue.cc:237] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:03.981382 6489 catalog_manager.cc:5484] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2). New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:14:03.991775 8329 raft_consensus.cc:482] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:03.992692 8329 raft_consensus.cc:504] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:03.994396 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b4e2d5fe4846e8b3c6081abb3c53da" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:14:03.994664 6768 raft_consensus.cc:2454] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:03.995151 6574 leader_election.cc:302] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:14:03.995179 8329 leader_election.cc:288] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:03.995393 8329 raft_consensus.cc:2781] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:03.995501 8329 raft_consensus.cc:482] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:03.995589 8329 raft_consensus.cc:3037] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:03.995654 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b4e2d5fe4846e8b3c6081abb3c53da" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:14:03.995896 6898 raft_consensus.cc:2454] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:03.997717 8329 raft_consensus.cc:504] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:03.998420 8329 leader_election.cc:288] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:03.998945 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b4e2d5fe4846e8b3c6081abb3c53da" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:03.998950 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b4e2d5fe4846e8b3c6081abb3c53da" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:03.999188 6768 raft_consensus.cc:3037] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:03.999205 6898 raft_consensus.cc:3037] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:04.001989 6768 raft_consensus.cc:2454] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:04.001986 6898 raft_consensus.cc:2454] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:04.002508 6571 leader_election.cc:302] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:04.002869 8329 raft_consensus.cc:2781] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:04.003062 8329 raft_consensus.cc:686] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:04.003369 8329 consensus_queue.cc:237] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:04.006522 6489 catalog_manager.cc:5484] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:04.357147 8066 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:04.363469 8308 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:04.370198 8308 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:04.394922 8308 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:04.411942 8122 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:04.414981 8333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:04.415453 8333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:14:04.420859 8122 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:04.425879 8333 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 299.54 ms
time per row: 0.14977 ms
I20240501 08:14:04.881494 6496 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42016:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:04.890848 6877 tablet_service.cc:1467] Processing CreateTablet for tablet cfc0c7b429754d92b4dd166c073ac8ed (DEFAULT_TABLE table=rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:04.891036 6618 tablet_service.cc:1467] Processing CreateTablet for tablet cfc0c7b429754d92b4dd166c073ac8ed (DEFAULT_TABLE table=rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:04.891367 6877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc0c7b429754d92b4dd166c073ac8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:04.891510 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc0c7b429754d92b4dd166c073ac8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:04.892138 6748 tablet_service.cc:1467] Processing CreateTablet for tablet 29506d751571401f86bced7e6f5415d7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:04.892220 7008 tablet_service.cc:1467] Processing CreateTablet for tablet cfc0c7b429754d92b4dd166c073ac8ed (DEFAULT_TABLE table=rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:04.892545 6748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29506d751571401f86bced7e6f5415d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:04.892614 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc0c7b429754d92b4dd166c073ac8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:04.895013 6616 tablet_service.cc:1467] Processing CreateTablet for tablet 049090443eaa4b18ba90cb60af9a87ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:04.895458 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049090443eaa4b18ba90cb60af9a87ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:04.898777 7007 tablet_service.cc:1467] Processing CreateTablet for tablet 049090443eaa4b18ba90cb60af9a87ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:04.899248 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049090443eaa4b18ba90cb60af9a87ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:04.906076 8405 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:04.907058 6878 tablet_service.cc:1467] Processing CreateTablet for tablet 049090443eaa4b18ba90cb60af9a87ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:04.907208 8406 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:04.907130 6879 tablet_service.cc:1467] Processing CreateTablet for tablet 29506d751571401f86bced7e6f5415d7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:04.907523 6878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049090443eaa4b18ba90cb60af9a87ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:04.908043 8405 tablet_bootstrap.cc:654] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:04.908174 6879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29506d751571401f86bced7e6f5415d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:04.908429 6616 tablet_service.cc:1467] Processing CreateTablet for tablet 29506d751571401f86bced7e6f5415d7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:04.912268 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29506d751571401f86bced7e6f5415d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:04.919163 8404 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:04.919350 8405 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:04.919611 8405 ts_tablet_manager.cc:1367] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240501 08:14:04.920162 8406 tablet_bootstrap.cc:654] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:04.920599 8405 raft_consensus.cc:348] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:04.920997 8405 raft_consensus.cc:374] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:04.921105 8404 tablet_bootstrap.cc:654] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:04.921140 8405 raft_consensus.cc:729] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:04.921535 8405 consensus_queue.cc:260] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:04.921787 8403 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:04.923502 8405 ts_tablet_manager.cc:1398] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:14:04.924196 8405 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:04.924882 8404 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c: No bootstrap required, opened a new log
I20240501 08:14:04.925015 8406 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:04.925100 8404 ts_tablet_manager.cc:1367] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 08:14:04.925240 8406 ts_tablet_manager.cc:1367] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 08:14:04.926106 8405 tablet_bootstrap.cc:654] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:04.926138 8406 raft_consensus.cc:348] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:04.926513 8406 raft_consensus.cc:374] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:04.926661 8406 raft_consensus.cc:729] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:04.927022 8403 tablet_bootstrap.cc:654] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:04.927018 8406 consensus_queue.cc:260] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:04.926069 8404 raft_consensus.cc:348] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:04.927295 8404 raft_consensus.cc:374] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:04.927407 8404 raft_consensus.cc:729] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:04.927670 8404 consensus_queue.cc:260] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:04.928748 8404 ts_tablet_manager.cc:1398] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:14:04.930964 8406 ts_tablet_manager.cc:1398] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:14:04.931222 8405 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:04.931417 8406 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:04.931442 8405 ts_tablet_manager.cc:1367] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240501 08:14:04.932951 8405 raft_consensus.cc:348] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:04.933243 8405 raft_consensus.cc:374] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:04.933326 8405 raft_consensus.cc:729] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:04.933516 8406 tablet_bootstrap.cc:654] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:04.933655 8405 consensus_queue.cc:260] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:04.936714 8405 ts_tablet_manager.cc:1398] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240501 08:14:04.937080 8405 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:14:04.939538 8403 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:04.939796 8403 ts_tablet_manager.cc:1367] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20240501 08:14:04.940212 8405 tablet_bootstrap.cc:654] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:04.941239 8406 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240: No bootstrap required, opened a new log
I20240501 08:14:04.941161 8403 raft_consensus.cc:348] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:04.941489 8406 ts_tablet_manager.cc:1367] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20240501 08:14:04.941478 8403 raft_consensus.cc:374] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:04.941624 8403 raft_consensus.cc:729] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:04.941972 8403 consensus_queue.cc:260] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:04.942327 8406 raft_consensus.cc:348] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:04.942615 8406 raft_consensus.cc:374] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:04.942713 8406 raft_consensus.cc:729] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:04.942973 8406 consensus_queue.cc:260] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:04.943531 8406 ts_tablet_manager.cc:1398] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:04.945572 8403 ts_tablet_manager.cc:1398] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:14:04.945945 8403 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:04.947057 8405 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699: No bootstrap required, opened a new log
I20240501 08:14:04.947211 8405 ts_tablet_manager.cc:1367] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240501 08:14:04.947633 8403 tablet_bootstrap.cc:654] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:04.947875 8405 raft_consensus.cc:348] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:04.948063 8405 raft_consensus.cc:374] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:04.948155 8405 raft_consensus.cc:729] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:14:04.948551 8405 consensus_queue.cc:260] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:04.949144 8405 ts_tablet_manager.cc:1398] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:14:04.949692 8403 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:04.949870 8403 ts_tablet_manager.cc:1367] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:14:04.950680 8403 raft_consensus.cc:348] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:04.950929 8403 raft_consensus.cc:374] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:04.951010 8403 raft_consensus.cc:729] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:04.951270 8403 consensus_queue.cc:260] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:04.951855 8403 ts_tablet_manager.cc:1398] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:04.952258 8403 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:04.953800 8403 tablet_bootstrap.cc:654] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:04.956946 8403 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac: No bootstrap required, opened a new log
I20240501 08:14:04.957105 8403 ts_tablet_manager.cc:1367] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:14:04.957791 8403 raft_consensus.cc:348] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:04.957983 8403 raft_consensus.cc:374] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:04.958076 8403 raft_consensus.cc:729] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:04.958312 8403 consensus_queue.cc:260] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:04.958822 8403 ts_tablet_manager.cc:1398] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:04.978117 8122 raft_consensus.cc:482] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:04.978384 8122 raft_consensus.cc:504] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:04.978999 8122 leader_election.cc:288] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:04.979614 6767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29506d751571401f86bced7e6f5415d7" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c" is_pre_election: true
I20240501 08:14:04.979617 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29506d751571401f86bced7e6f5415d7" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:14:04.979864 6898 raft_consensus.cc:2454] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:04.979864 6767 raft_consensus.cc:2454] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:04.980357 6571 leader_election.cc:302] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:04.980674 8122 raft_consensus.cc:2781] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:04.980769 8122 raft_consensus.cc:482] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:04.980870 8122 raft_consensus.cc:3037] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:04.983357 8122 raft_consensus.cc:504] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:04.984026 8122 leader_election.cc:288] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:04.984593 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29506d751571401f86bced7e6f5415d7" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:04.984592 6767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29506d751571401f86bced7e6f5415d7" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:04.984812 6767 raft_consensus.cc:3037] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:04.984812 6898 raft_consensus.cc:3037] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:04.987707 6767 raft_consensus.cc:2454] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:04.987707 6898 raft_consensus.cc:2454] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:04.988202 6571 leader_election.cc:302] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:04.988559 8122 raft_consensus.cc:2781] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:04.988730 8122 raft_consensus.cc:686] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:04.989003 8122 consensus_queue.cc:237] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:14:04.992307 6491 catalog_manager.cc:5484] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:14:05.028841 8122 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:05.029063 8122 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:05.030483 8122 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:05.030493 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049090443eaa4b18ba90cb60af9a87ab" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:14:05.030781 7028 raft_consensus.cc:2454] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:05.030959 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049090443eaa4b18ba90cb60af9a87ab" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" is_pre_election: true
I20240501 08:14:05.031247 6898 raft_consensus.cc:2454] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 0.
I20240501 08:14:05.031445 6573 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:14:05.031786 8122 raft_consensus.cc:2781] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:05.031919 8122 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:05.032019 8122 raft_consensus.cc:3037] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:05.034288 8122 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:05.035063 8122 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:05.035871 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049090443eaa4b18ba90cb60af9a87ab" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:05.035856 6898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049090443eaa4b18ba90cb60af9a87ab" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:05.036084 7028 raft_consensus.cc:3037] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:05.036209 6898 raft_consensus.cc:3037] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:05.037273 8338 raft_consensus.cc:482] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:05.037467 8338 raft_consensus.cc:504] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:05.038468 8338 leader_election.cc:288] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:05.039058 7028 raft_consensus.cc:2454] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:05.039075 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc0c7b429754d92b4dd166c073ac8ed" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:14:05.039212 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc0c7b429754d92b4dd166c073ac8ed" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
I20240501 08:14:05.039340 6638 raft_consensus.cc:2454] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 0.
I20240501 08:14:05.039422 7029 raft_consensus.cc:2454] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 0.
I20240501 08:14:05.039968 6834 leader_election.cc:302] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:05.040118 6573 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:14:05.040441 8338 raft_consensus.cc:2781] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:05.040563 8338 raft_consensus.cc:482] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:05.040647 8338 raft_consensus.cc:3037] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:05.040653 8122 raft_consensus.cc:2781] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:05.040777 6898 raft_consensus.cc:2454] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 1.
I20240501 08:14:05.041159 8122 raft_consensus.cc:686] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:05.041501 8122 consensus_queue.cc:237] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:05.043644 8338 raft_consensus.cc:504] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:05.044400 8338 leader_election.cc:288] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 election: Requested vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:05.044989 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc0c7b429754d92b4dd166c073ac8ed" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:05.045239 6638 raft_consensus.cc:3037] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:05.045255 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc0c7b429754d92b4dd166c073ac8ed" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:05.045495 7028 raft_consensus.cc:3037] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:05.048606 7028 raft_consensus.cc:2454] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 1.
I20240501 08:14:05.048615 6638 raft_consensus.cc:2454] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 1.
I20240501 08:14:05.049233 6834 leader_election.cc:302] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:05.049758 8338 raft_consensus.cc:2781] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:05.051759 8338 raft_consensus.cc:686] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Becoming Leader. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Running, Role: LEADER
I20240501 08:14:05.052145 8338 consensus_queue.cc:237] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:05.064303 6491 catalog_manager.cc:5484] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:05.066620 6496 catalog_manager.cc:5484] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: term changed from 0 to 1, leader changed from <none> to e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3). New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
W20240501 08:14:05.477686 8420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:05.478091 8420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:05.486101 8420 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:14:05.491861 8122 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:05.497339 8347 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:05.533684 8347 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:05.538872 8122 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240501 08:14:05.553404 8338 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:05.565784 8338 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:05.725884 6574 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 47622.2us
INSERT report
rows total: 2000
time total: 228.814 ms
time per row: 0.114407 ms
I20240501 08:14:05.907759 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.5:45691
--local_ip_for_outbound_sockets=127.25.107.5
--tserver_master_addrs=127.25.107.62:36223
--webserver_port=37091
--webserver_interface=127.25.107.5
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:14:06.299531 8481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:06.299834 8481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:06.299964 8481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:06.308413 8481 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:06.308558 8481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:06.308719 8481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.5
I20240501 08:14:06.318861 8481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.5:45691
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.107.5
--webserver_port=37091
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.5
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:06.320348 8481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:06.322171 8481 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:06.335301 8487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:06.335376 8486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:06.335397 8489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:06.336167 8481 server_base.cc:1034] running on GCE node
I20240501 08:14:06.336819 8481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:06.337857 8481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:06.339098 8481 hybrid_clock.cc:648] HybridClock initialized: now 1714551246339052 us; error 54 us; skew 500 ppm
I20240501 08:14:06.339455 8481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:06.341939 8481 webserver.cc:460] Webserver started at http://127.25.107.5:37091/ using document root <none> and password file <none>
I20240501 08:14:06.342800 8481 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:06.342943 8481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:06.348737 8481 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 08:14:06.351949 8495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:06.353807 8481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240501 08:14:06.353996 8481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0a814bba1abe4653bcd418a83ec42370"
format_stamp: "Formatted at 2024-05-01 08:13:47 on dist-test-slave-dv41"
I20240501 08:14:06.354974 8481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:06.378172 8481 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:06.379119 8481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:06.379419 8481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:06.380378 8481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:06.382004 8481 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:06.382086 8481 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:06.382216 8481 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:06.382288 8481 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240501 08:14:06.423545 8481 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.5:45691
I20240501 08:14:06.423581 8607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.5:45691 every 8 connection(s)
I20240501 08:14:06.425143 8481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:14:06.432973 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 8481
I20240501 08:14:06.433753 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.6:41219
--local_ip_for_outbound_sockets=127.25.107.6
--tserver_master_addrs=127.25.107.62:36223
--webserver_port=33059
--webserver_interface=127.25.107.6
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:06.442385 8608 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:14:06.442783 8608 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:06.443606 8608 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:14:06.445358 6488 ts_manager.cc:194] Re-registered known tserver with Master: 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:14:06.446501 6488 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.5:36249
W20240501 08:14:06.823433 8612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:06.823737 8612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:06.823872 8612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:06.832487 8612 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:06.832650 8612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:06.832814 8612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.6
I20240501 08:14:06.843333 8612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.6:41219
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.107.6
--webserver_port=33059
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.6
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:06.844732 8612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:06.846568 8612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:06.859146 8618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:06.859190 8619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:06.859789 8621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:06.860671 8612 server_base.cc:1034] running on GCE node
I20240501 08:14:06.861443 8612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:06.862656 8612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:06.863947 8612 hybrid_clock.cc:648] HybridClock initialized: now 1714551246863876 us; error 64 us; skew 500 ppm
I20240501 08:14:06.864257 8612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:06.866731 8612 webserver.cc:460] Webserver started at http://127.25.107.6:33059/ using document root <none> and password file <none>
I20240501 08:14:06.867482 8612 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:06.867600 8612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:06.873569 8612 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:14:06.876925 8627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:06.878700 8612 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 08:14:06.878894 8612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "aba7c226aaca4e8c90931f547d48aaa3"
format_stamp: "Formatted at 2024-05-01 08:13:48 on dist-test-slave-dv41"
I20240501 08:14:06.879757 8612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:06.899374 8612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:06.900310 8612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:06.900592 8612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:06.901676 8612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:06.903429 8612 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:06.903509 8612 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:06.903635 8612 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:06.903684 8612 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:06.934015 8612 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.6:41219
I20240501 08:14:06.934080 8739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.6:41219 every 8 connection(s)
I20240501 08:14:06.935655 8612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:14:06.937167 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 8612
I20240501 08:14:06.937772 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.7:33929
--local_ip_for_outbound_sockets=127.25.107.7
--tserver_master_addrs=127.25.107.62:36223
--webserver_port=35309
--webserver_interface=127.25.107.7
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:06.955956 8740 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:14:06.956338 8740 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:06.957376 8740 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:14:06.959436 6488 ts_manager.cc:194] Re-registered known tserver with Master: aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
I20240501 08:14:06.960670 6488 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.6:52133
W20240501 08:14:07.327301 8744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:07.327628 8744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:07.327767 8744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:07.336239 8744 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:07.336397 8744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:07.336548 8744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.7
I20240501 08:14:07.346166 8744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.7:33929
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.107.7
--webserver_port=35309
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.7
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:07.347569 8744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:07.349467 8744 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:07.362402 8750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:07.362592 8753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:07.362875 8751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:07.363632 8744 server_base.cc:1034] running on GCE node
I20240501 08:14:07.364388 8744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:07.365448 8744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:07.366643 8744 hybrid_clock.cc:648] HybridClock initialized: now 1714551247366589 us; error 49 us; skew 500 ppm
I20240501 08:14:07.366971 8744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:07.369279 8744 webserver.cc:460] Webserver started at http://127.25.107.7:35309/ using document root <none> and password file <none>
I20240501 08:14:07.370067 8744 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:07.370203 8744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:07.376032 8744 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 08:14:07.379177 8759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:07.380640 8744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:14:07.380824 8744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a518ca5e9de24204bd028b0ce4a77d58"
format_stamp: "Formatted at 2024-05-01 08:13:49 on dist-test-slave-dv41"
I20240501 08:14:07.381775 8744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:07.403389 8744 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:07.404312 8744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:07.404616 8744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:07.406193 8744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:07.408650 8744 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:07.408751 8744 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:07.408885 8744 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:07.408957 8744 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:07.449297 8608 heartbeater.cc:503] Master 127.25.107.62:36223 was elected leader, sending a full tablet report...
I20240501 08:14:07.450639 8744 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.7:33929
I20240501 08:14:07.450685 8871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.7:33929 every 8 connection(s)
I20240501 08:14:07.452471 8744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:14:07.462383 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 8744
I20240501 08:14:07.463193 26028 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
/tmp/dist-test-taskDw_B8y/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.107.8:41625
--local_ip_for_outbound_sockets=127.25.107.8
--tserver_master_addrs=127.25.107.62:36223
--webserver_port=37217
--webserver_interface=127.25.107.8
--builtin_ntp_servers=127.25.107.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:07.466667 8872 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:14:07.467093 8872 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:07.468060 8872 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:14:07.469729 6488 ts_manager.cc:194] Re-registered known tserver with Master: a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929)
I20240501 08:14:07.470626 6488 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.7:46741
W20240501 08:14:07.870652 8876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:07.870960 8876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:07.871078 8876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:07.878894 8876 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:07.879034 8876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:07.879209 8876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.107.8
I20240501 08:14:07.889794 8876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.107.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.107.8:41625
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.25.107.8
--webserver_port=37217
--tserver_master_addrs=127.25.107.62:36223
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.107.8
--log_dir=/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:07.891391 8876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:07.893440 8876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:07.906355 8885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:07.906384 8883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:07.907140 8882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:07.907316 8876 server_base.cc:1034] running on GCE node
I20240501 08:14:07.908473 8876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:07.909569 8876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:07.910806 8876 hybrid_clock.cc:648] HybridClock initialized: now 1714551247910747 us; error 51 us; skew 500 ppm
I20240501 08:14:07.911191 8876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:07.913897 8876 webserver.cc:460] Webserver started at http://127.25.107.8:37217/ using document root <none> and password file <none>
I20240501 08:14:07.914800 8876 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:07.914950 8876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:07.921684 8876 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:14:07.924822 8891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:07.926470 8876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:14:07.926676 8876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "19fdb76c8d1f41d7842e397adfc1ef3f"
format_stamp: "Formatted at 2024-05-01 08:13:49 on dist-test-slave-dv41"
I20240501 08:14:07.927556 8876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:07.956032 8876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:07.957087 8876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:07.957372 8876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:07.958549 8876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:07.960309 8876 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:07.960395 8876 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:07.960520 8876 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:07.960570 8876 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:07.964035 8740 heartbeater.cc:503] Master 127.25.107.62:36223 was elected leader, sending a full tablet report...
I20240501 08:14:07.999631 8876 rpc_server.cc:307] RPC server started. Bound to: 127.25.107.8:41625
I20240501 08:14:07.999687 9003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.107.8:41625 every 8 connection(s)
I20240501 08:14:08.001260 8876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:14:08.007388 26028 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu as pid 8876
I20240501 08:14:08.024070 9004 heartbeater.cc:347] Connected to a master server at 127.25.107.62:36223
I20240501 08:14:08.024467 9004 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:08.025518 9004 heartbeater.cc:511] Master 127.25.107.62:36223 requested a full tablet report, sending...
I20240501 08:14:08.027365 6488 ts_manager.cc:194] Re-registered known tserver with Master: 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625)
I20240501 08:14:08.028479 6488 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.107.8:44697
I20240501 08:14:08.473839 8872 heartbeater.cc:503] Master 127.25.107.62:36223 was elected leader, sending a full tablet report...
I20240501 08:14:09.033358 9004 heartbeater.cc:503] Master 127.25.107.62:36223 was elected leader, sending a full tablet report...
I20240501 08:14:09.344275 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:09.366905 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:09.381314 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:09.432413 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:09.480480 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:09.493600 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:09.499017 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:09.499631 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:09.518132 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:09.531165 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:09.540307 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:09.569710 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:09.570549 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:09.604724 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:09.619380 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:09.659425 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:10.216193 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.217480 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.254494 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.295931 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:10.296679 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.327097 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.354730 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:10.385406 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.391310 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.416692 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:10.419744 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:10.439621 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:10.445209 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:10.446090 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.456794 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:10.468814 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.498724 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.514627 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:10.545117 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.551122 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:10.560532 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:10.610376 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:10.611438 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:10.646648 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:11.088025 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:11.093479 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:11.124416 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:11.141947 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:11.225373 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:11.235832 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:11.258548 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:11.269604 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:11.271533 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:11.273826 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:11.309350 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:11.319342 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:11.320004 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:11.336499 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:11.361440 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:11.394450 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:12.113307 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.131501 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:12.134450 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.216861 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:12.249634 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.319787 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.354490 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:12.361769 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:12.372637 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:12.381881 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:12.393455 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.411043 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:12.412572 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:12.436482 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.456287 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.456804 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.462344 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.470723 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.480636 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:12.490597 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:12.499907 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.552624 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.593708 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:12.596352 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:12.721745 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.728876 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.735544 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:12.743530 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:12.763082 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:12.804179 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:12.811372 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:12.832595 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:13.220403 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:13.221768 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:13.263525 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:13.314844 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:13.349997 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:14:13.365636 6962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.3:43505, user_credentials={real_user=slave}} blocked reactor thread for 55408.9us
I20240501 08:14:13.365995 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:13.393371 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:13.414014 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:13.449728 7029 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:13.493858 6637 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:13.493858 6899 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:13.494921 9662 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:13.495874 9662 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:14:13.512954 6965 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f090fda814f455a88dccd6680bdd05f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:13.528592 9500 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:13.532109 6899 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:13.545640 6637 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:13.546557 6488 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
W20240501 08:14:13.571708 6573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 51161.5us
I20240501 08:14:13.789367 6898 consensus_queue.cc:237] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:13.809973 6767 raft_consensus.cc:1261] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:13.811769 9217 consensus_queue.cc:1035] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:13.835388 7029 raft_consensus.cc:1261] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:13.836254 9217 consensus_queue.cc:1035] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:13.877839 9757 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:13.884459 7029 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:13.898574 6491 catalog_manager.cc:5484] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:13.902523 6767 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:14:13.966089 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6575b94bd51546f1bd3bb862f012e7ca. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:13.972963 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.001024 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.025661 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:14.059257 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.069283 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:14.095690 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.120604 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:14.160714 7029 consensus_queue.cc:237] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:14.167946 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:14.181958 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.184388 9818 ts_tablet_manager.cc:902] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:14.191413 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:14:14.199429 6962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.1:45193, user_credentials={real_user=slave}} blocked reactor thread for 95570.2us
I20240501 08:14:14.201172 6637 raft_consensus.cc:1261] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:14.215572 9500 consensus_queue.cc:1035] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:14.240784 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:14.245527 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:14.260862 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:14.271417 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:14:14.282301 6963 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b7f87b6032d45c5be5d748ab2cc71e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:14.288367 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.297161 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.319250 9842 ts_tablet_manager.cc:902] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:14.326001 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.338033 6768 raft_consensus.cc:1261] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:14.339162 9813 consensus_queue.cc:1035] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:14.352643 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.357565 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.363971 6637 consensus_queue.cc:237] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:14.373963 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:14.387223 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:14.420922 9827 raft_consensus.cc:2932] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:14.427203 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:14.436173 6767 raft_consensus.cc:2932] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:14.451336 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:14.454631 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:14.462388 6491 catalog_manager.cc:5484] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:14.474807 6638 raft_consensus.cc:2932] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:14.485630 9818 tablet_copy_client.cc:308] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:14.492498 6898 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:14.493585 9697 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:14.498039 9842 tablet_copy_client.cc:308] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:14:14.508395 6768 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:14.509305 9852 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:14:14.522076 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13b4e2d5fe4846e8b3c6081abb3c53da. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:14.528115 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 5f090fda814f455a88dccd6680bdd05f from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:34223)
I20240501 08:14:14.528471 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 5f090fda814f455a88dccd6680bdd05f from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:34223: session id = 0a814bba1abe4653bcd418a83ec42370-5f090fda814f455a88dccd6680bdd05f
I20240501 08:14:14.532397 9697 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:14.533735 7049 tablet_copy_source_session.cc:205] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:14.533454 6767 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:14.540643 9818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f090fda814f455a88dccd6680bdd05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:14.550570 6488 catalog_manager.cc:5484] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:14.571574 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 6575b94bd51546f1bd3bb862f012e7ca from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:40161)
I20240501 08:14:14.572062 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 6575b94bd51546f1bd3bb862f012e7ca from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:40161: session id = aba7c226aaca4e8c90931f547d48aaa3-6575b94bd51546f1bd3bb862f012e7ca
I20240501 08:14:14.576709 6919 tablet_copy_source_session.cc:205] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:14.584604 6898 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:14.587911 9842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6575b94bd51546f1bd3bb862f012e7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:14.689165 6637 consensus_queue.cc:237] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:14.689695 9818 tablet_copy_client.cc:783] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:14.690470 9818 tablet_copy_client.cc:647] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:14.702008 9893 ts_tablet_manager.cc:902] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:14.703676 9818 tablet_copy_client.cc:523] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:14.706045 6767 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:14.706243 7029 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:14.706844 9874 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:14.707834 9874 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:14.733160 9818 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:14.738075 9874 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:14.742805 7029 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:14.748064 6767 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:14.750511 6488 catalog_manager.cc:5484] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:14.752290 9842 tablet_copy_client.cc:783] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:14.752728 9842 tablet_copy_client.cc:647] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
W20240501 08:14:14.755509 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 491674dfbf74498e8a787bebaa5e95da. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:14.757892 9842 tablet_copy_client.cc:523] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:14.770354 9842 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:14.786926 9893 tablet_copy_client.cc:308] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:14.796468 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 8b7f87b6032d45c5be5d748ab2cc71e2 from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:46523)
I20240501 08:14:14.796759 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 8b7f87b6032d45c5be5d748ab2cc71e2 from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:46523: session id = aba7c226aaca4e8c90931f547d48aaa3-8b7f87b6032d45c5be5d748ab2cc71e2
I20240501 08:14:14.800314 7049 tablet_copy_source_session.cc:205] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:14.802505 9893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7f87b6032d45c5be5d748ab2cc71e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:14.815268 9893 tablet_copy_client.cc:783] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:14.836412 7029 consensus_queue.cc:237] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:14.842152 6896 raft_consensus.cc:1261] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:14.842473 6637 raft_consensus.cc:1261] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:14.843446 9813 consensus_queue.cc:1035] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:14.843538 9893 tablet_copy_client.cc:647] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:14.844496 9827 consensus_queue.cc:1035] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:14.846331 9893 tablet_copy_client.cc:523] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:14.851367 9893 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:14.855875 9818 log.cc:824] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: Log is configured to *not* fsync() on all Append() calls
I20240501 08:14:14.864538 9893 log.cc:824] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: Log is configured to *not* fsync() on all Append() calls
I20240501 08:14:14.879293 9827 raft_consensus.cc:2932] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:14.883762 6638 raft_consensus.cc:2932] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:14.889741 6896 raft_consensus.cc:2932] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:14.891530 6496 catalog_manager.cc:5484] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:14:14.909482 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a6efa669c1645aaa971d59e5d38c8d5. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:14.993016 6638 consensus_queue.cc:237] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:14.997937 6896 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:14.998474 6767 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:14:14.998718 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13b4e2d5fe4846e8b3c6081abb3c53da. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:14.999248 9697 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:15.002384 9697 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:15.006985 9697 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:15.011184 6767 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:14:15.021395 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13b4e2d5fe4846e8b3c6081abb3c53da. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:15.024991 9818 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:15.024700 6496 catalog_manager.cc:5484] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:15.025768 9818 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:15.026361 9818 ts_tablet_manager.cc:1367] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.293s user 0.068s sys 0.004s
I20240501 08:14:15.033592 9818 raft_consensus.cc:348] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:15.034406 9818 raft_consensus.cc:729] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:15.036353 6896 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:15.038652 9818 consensus_queue.cc:260] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:15.102569 9818 ts_tablet_manager.cc:1398] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.076s user 0.032s sys 0.028s
I20240501 08:14:15.106572 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-5f090fda814f455a88dccd6680bdd05f received from {username='slave'} at 127.25.107.5:34223
I20240501 08:14:15.106788 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-5f090fda814f455a88dccd6680bdd05f on tablet 5f090fda814f455a88dccd6680bdd05f with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:15.122946 9818 ts_tablet_manager.cc:902] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:15.144147 6896 consensus_queue.cc:237] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:15.163398 9818 tablet_copy_client.cc:308] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:15.168615 9842 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:15.169333 9842 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:15.169971 9842 ts_tablet_manager.cc:1367] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.400s user 0.063s sys 0.004s
I20240501 08:14:15.180899 6638 raft_consensus.cc:1261] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:15.181731 9757 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:15.188890 9842 raft_consensus.cc:348] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:15.189648 9842 raft_consensus.cc:729] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:15.191143 9842 consensus_queue.cc:260] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:15.192291 7029 raft_consensus.cc:1261] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:15.193010 9757 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:15.219879 9842 ts_tablet_manager.cc:1398] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.050s user 0.011s sys 0.008s
I20240501 08:14:15.222070 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-6575b94bd51546f1bd3bb862f012e7ca received from {username='slave'} at 127.25.107.6:40161
I20240501 08:14:15.222316 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-6575b94bd51546f1bd3bb862f012e7ca on tablet 6575b94bd51546f1bd3bb862f012e7ca with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:15.227501 9757 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:15.228704 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 491674dfbf74498e8a787bebaa5e95da from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:51915)
I20240501 08:14:15.229010 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 491674dfbf74498e8a787bebaa5e95da from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:51915: session id = 0a814bba1abe4653bcd418a83ec42370-491674dfbf74498e8a787bebaa5e95da
I20240501 08:14:15.232255 6637 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:15.234791 6658 tablet_copy_source_session.cc:205] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:15.238092 9893 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:15.238602 9893 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:15.238883 9893 ts_tablet_manager.cc:1367] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.388s user 0.055s sys 0.010s
W20240501 08:14:15.240052 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfc0c7b429754d92b4dd166c073ac8ed. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:15.240479 9893 raft_consensus.cc:348] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:15.240717 9893 raft_consensus.cc:729] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:15.241629 9893 consensus_queue.cc:260] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:15.243250 8563 raft_consensus.cc:1203] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:15.245694 9893 ts_tablet_manager.cc:1398] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 08:14:15.248914 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-8b7f87b6032d45c5be5d748ab2cc71e2 received from {username='slave'} at 127.25.107.6:46523
I20240501 08:14:15.249064 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-8b7f87b6032d45c5be5d748ab2cc71e2 on tablet 8b7f87b6032d45c5be5d748ab2cc71e2 with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:15.253338 6491 catalog_manager.cc:5484] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:15.256042 7027 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:15.270778 9818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491674dfbf74498e8a787bebaa5e95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:15.277941 8695 raft_consensus.cc:1203] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:15.287061 9818 tablet_copy_client.cc:783] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:15.288066 9818 tablet_copy_client.cc:647] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:15.291775 9818 tablet_copy_client.cc:523] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:15.310187 9818 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:15.310544 6638 consensus_queue.cc:237] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
W20240501 08:14:15.326856 6571 consensus_peers.cc:485] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 070e927b5d684367ae0b40b12a23da54. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:15.328524 6896 raft_consensus.cc:1261] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:15.329183 9697 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:15.336925 6767 raft_consensus.cc:1261] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:15.337682 9697 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:15.345420 7027 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:15.352962 6768 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:15.353044 6896 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:15.354064 9813 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 08:14:15.356353 9827 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:15.362954 9697 raft_consensus.cc:2932] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:15.367594 6769 raft_consensus.cc:2932] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:15.374423 6898 raft_consensus.cc:2932] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:15.376705 6488 catalog_manager.cc:5484] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:15.421375 10044 ts_tablet_manager.cc:902] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:15.423763 9827 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:15.427353 6898 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:15.443974 6768 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:15.446969 6488 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:15.454941 10049 ts_tablet_manager.cc:902] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:15.502337 10044 tablet_copy_client.cc:308] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
W20240501 08:14:15.532985 6962 consensus_peers.cc:485] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57ecbb30f7764fceb008baaa2c37da8c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:15.546940 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 13b4e2d5fe4846e8b3c6081abb3c53da from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:48573)
I20240501 08:14:15.547159 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 13b4e2d5fe4846e8b3c6081abb3c53da from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:48573: session id = a518ca5e9de24204bd028b0ce4a77d58-13b4e2d5fe4846e8b3c6081abb3c53da
I20240501 08:14:15.550493 6658 tablet_copy_source_session.cc:205] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:15.557071 10044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b4e2d5fe4846e8b3c6081abb3c53da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:15.574199 10073 ts_tablet_manager.cc:902] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:15.579952 10049 tablet_copy_client.cc:308] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:15.607699 9818 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:15.608173 9818 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:15.608474 9818 ts_tablet_manager.cc:1367] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.302s user 0.048s sys 0.027s
I20240501 08:14:15.609508 9818 raft_consensus.cc:348] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:15.609797 9818 raft_consensus.cc:729] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:15.610040 9818 consensus_queue.cc:260] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:15.613435 9818 ts_tablet_manager.cc:1398] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:14:15.639649 10097 ts_tablet_manager.cc:902] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:15.658221 10044 tablet_copy_client.cc:783] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:15.659927 10044 tablet_copy_client.cc:647] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:15.661058 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:15.664670 10044 tablet_copy_client.cc:523] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:15.673174 10044 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:15.678740 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:15.688767 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:15.695643 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:15.703783 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-491674dfbf74498e8a787bebaa5e95da received from {username='slave'} at 127.25.107.5:51915
I20240501 08:14:15.703996 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-491674dfbf74498e8a787bebaa5e95da on tablet 491674dfbf74498e8a787bebaa5e95da with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:15.710278 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 4a6efa669c1645aaa971d59e5d38c8d5 from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:37833)
I20240501 08:14:15.710480 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 4a6efa669c1645aaa971d59e5d38c8d5 from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:37833: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-4a6efa669c1645aaa971d59e5d38c8d5
I20240501 08:14:15.713578 7049 tablet_copy_source_session.cc:205] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:15.714298 10073 tablet_copy_client.cc:308] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:15.715167 8695 raft_consensus.cc:1203] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:15.721876 10049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6efa669c1645aaa971d59e5d38c8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:15.740263 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 13b4e2d5fe4846e8b3c6081abb3c53da from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:58105)
I20240501 08:14:15.740484 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 13b4e2d5fe4846e8b3c6081abb3c53da from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:58105: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-13b4e2d5fe4846e8b3c6081abb3c53da
I20240501 08:14:15.743636 6658 tablet_copy_source_session.cc:205] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:15.745715 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b4e2d5fe4846e8b3c6081abb3c53da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:15.748008 6638 consensus_queue.cc:237] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
W20240501 08:14:15.760069 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 491674dfbf74498e8a787bebaa5e95da. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:15.761590 8562 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:15.762485 9697 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:15.770430 7027 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:15.771466 6768 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:15.771888 9872 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:15.772575 9872 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:15.780020 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:15.780239 10097 tablet_copy_client.cc:308] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:14:15.812057 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:15.816187 10049 tablet_copy_client.cc:783] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:15.816644 10073 tablet_copy_client.cc:783] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:15.816931 10073 tablet_copy_client.cc:647] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:15.817152 10049 tablet_copy_client.cc:647] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:15.818862 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:15.820816 10049 tablet_copy_client.cc:523] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:15.821738 6896 consensus_queue.cc:237] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:15.822952 10073 tablet_copy_client.cc:523] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 08:14:15.825103 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfc0c7b429754d92b4dd166c073ac8ed. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:15.825978 7029 raft_consensus.cc:1261] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:15.826735 10003 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:15.830951 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet cfc0c7b429754d92b4dd166c073ac8ed from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:46197)
I20240501 08:14:15.831182 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet cfc0c7b429754d92b4dd166c073ac8ed from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:46197: session id = 0a814bba1abe4653bcd418a83ec42370-cfc0c7b429754d92b4dd166c073ac8ed
W20240501 08:14:15.834880 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfc0c7b429754d92b4dd166c073ac8ed. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:15.835888 9872 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:15.837116 10049 tablet_bootstrap.cc:492] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:15.837857 6919 tablet_copy_source_session.cc:205] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:15.839601 10097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc0c7b429754d92b4dd166c073ac8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:15.839646 6638 raft_consensus.cc:1261] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:15.840373 10003 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:15.842597 6768 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:15.843290 9757 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:15.847160 10097 tablet_copy_client.cc:783] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:15.847103 6637 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:15.847960 10097 tablet_copy_client.cc:647] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:15.848850 6496 catalog_manager.cc:5484] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:15.849550 10073 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:15.853130 10097 tablet_copy_client.cc:523] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:15.856920 8562 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:15.859385 7029 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:15.862352 6496 catalog_manager.cc:5484] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:15.868439 10097 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:15.875718 10127 ts_tablet_manager.cc:902] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:15.880077 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:15.854841 7027 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:15.940598 10152 ts_tablet_manager.cc:902] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:15.996512 10044 log.cc:824] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: Log is configured to *not* fsync() on all Append() calls
I20240501 08:14:16.000660 10049 log.cc:824] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: Log is configured to *not* fsync() on all Append() calls
I20240501 08:14:16.038797 10127 tablet_copy_client.cc:308] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:16.039793 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 070e927b5d684367ae0b40b12a23da54 from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:48573)
I20240501 08:14:16.040025 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 070e927b5d684367ae0b40b12a23da54 from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:48573: session id = a518ca5e9de24204bd028b0ce4a77d58-070e927b5d684367ae0b40b12a23da54
I20240501 08:14:16.043391 6658 tablet_copy_source_session.cc:205] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:16.046880 10127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070e927b5d684367ae0b40b12a23da54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:16.082564 10152 tablet_copy_client.cc:308] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:16.095989 10127 tablet_copy_client.cc:783] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:16.098711 10127 tablet_copy_client.cc:647] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:16.104179 10127 tablet_copy_client.cc:523] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:16.108194 10127 tablet_bootstrap.cc:492] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:16.149560 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 57ecbb30f7764fceb008baaa2c37da8c from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:54455)
I20240501 08:14:16.149796 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 57ecbb30f7764fceb008baaa2c37da8c from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:54455: session id = a518ca5e9de24204bd028b0ce4a77d58-57ecbb30f7764fceb008baaa2c37da8c
I20240501 08:14:16.153051 7049 tablet_copy_source_session.cc:205] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:16.155267 10152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ecbb30f7764fceb008baaa2c37da8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:16.164629 10152 tablet_copy_client.cc:783] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:16.165032 10152 tablet_copy_client.cc:647] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:16.167215 10152 tablet_copy_client.cc:523] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:16.171098 10152 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:16.173146 10097 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:14:16.173835 10097 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:16.174171 10097 ts_tablet_manager.cc:1367] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.306s user 0.047s sys 0.011s
I20240501 08:14:16.175061 10097 raft_consensus.cc:348] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:16.175633 10097 raft_consensus.cc:729] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:16.175854 10097 consensus_queue.cc:260] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:16.176546 10097 ts_tablet_manager.cc:1398] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:16.179097 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-cfc0c7b429754d92b4dd166c073ac8ed received from {username='slave'} at 127.25.107.5:46197
I20240501 08:14:16.179248 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-cfc0c7b429754d92b4dd166c073ac8ed on tablet cfc0c7b429754d92b4dd166c073ac8ed with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:16.223870 10044 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:16.224568 10044 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:16.225206 10044 ts_tablet_manager.cc:1367] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.552s user 0.058s sys 0.016s
I20240501 08:14:16.240901 10044 raft_consensus.cc:348] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:16.241564 10044 raft_consensus.cc:729] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:16.246302 10044 consensus_queue.cc:260] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:16.249837 6768 consensus_queue.cc:237] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:16.255497 6896 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:16.256417 10117 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:16.257270 10073 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:16.264850 10044 ts_tablet_manager.cc:1398] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.040s user 0.008s sys 0.007s
I20240501 08:14:16.270984 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-13b4e2d5fe4846e8b3c6081abb3c53da received from {username='slave'} at 127.25.107.7:48573
I20240501 08:14:16.271168 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-13b4e2d5fe4846e8b3c6081abb3c53da on tablet 13b4e2d5fe4846e8b3c6081abb3c53da with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:16.275192 7029 consensus_queue.cc:237] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:16.278206 10073 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:16.278980 10073 ts_tablet_manager.cc:1367] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.430s user 0.059s sys 0.009s
I20240501 08:14:16.286302 10231 ts_tablet_manager.cc:902] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:16.287015 10073 raft_consensus.cc:348] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:16.287730 10073 raft_consensus.cc:729] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:16.288679 10073 consensus_queue.cc:260] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
W20240501 08:14:16.289754 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60238162349647c681685349c6f8ede4. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:16.295552 6705 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7903106d7e74e1ea102b1abdbfce63e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:16.313374 6638 raft_consensus.cc:1261] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:16.315425 6896 raft_consensus.cc:1261] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:16.317569 7029 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:16.318208 9827 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:16.319106 9827 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:16.325217 10231 tablet_copy_client.cc:308] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:16.326181 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 491674dfbf74498e8a787bebaa5e95da from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:58105)
I20240501 08:14:16.327379 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 491674dfbf74498e8a787bebaa5e95da from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:58105: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-491674dfbf74498e8a787bebaa5e95da
I20240501 08:14:16.330739 6658 tablet_copy_source_session.cc:205] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:16.332424 10231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491674dfbf74498e8a787bebaa5e95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:16.339818 10127 tablet_bootstrap.cc:492] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:16.340354 10127 tablet_bootstrap.cc:492] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:16.340662 10127 ts_tablet_manager.cc:1367] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.233s user 0.052s sys 0.001s
I20240501 08:14:16.341882 10127 raft_consensus.cc:348] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:16.342156 10127 raft_consensus.cc:729] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:16.342460 10127 consensus_queue.cc:260] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:16.343029 10127 ts_tablet_manager.cc:1398] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:16.343988 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-070e927b5d684367ae0b40b12a23da54 received from {username='slave'} at 127.25.107.7:48573
I20240501 08:14:16.344167 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-070e927b5d684367ae0b40b12a23da54 on tablet 070e927b5d684367ae0b40b12a23da54 with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:16.347658 10231 tablet_copy_client.cc:783] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:16.348012 10231 tablet_copy_client.cc:647] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:16.349985 10231 tablet_copy_client.cc:523] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:16.354130 10231 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:16.356098 10117 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:16.361719 10243 ts_tablet_manager.cc:902] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:16.380373 10152 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:16.380937 10152 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:16.381284 10152 ts_tablet_manager.cc:1367] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.210s user 0.071s sys 0.048s
I20240501 08:14:16.382099 10152 raft_consensus.cc:348] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:16.382429 10152 raft_consensus.cc:729] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:16.382750 10152 consensus_queue.cc:260] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:16.384083 10152 ts_tablet_manager.cc:1398] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:14:16.384886 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-57ecbb30f7764fceb008baaa2c37da8c received from {username='slave'} at 127.25.107.7:54455
I20240501 08:14:16.385033 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-57ecbb30f7764fceb008baaa2c37da8c on tablet 57ecbb30f7764fceb008baaa2c37da8c with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:16.410938 10097 tablet_replica.cc:330] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: stopping tablet replica
I20240501 08:14:16.411559 10097 raft_consensus.cc:2229] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:16.411897 10097 pending_rounds.cc:62] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Trying to abort 1 pending ops.
I20240501 08:14:16.412026 10097 pending_rounds.cc:68] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022801943843586048 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "cfc0c7b429754d92b4dd166c073ac8ed" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } } }
I20240501 08:14:16.412436 10097 raft_consensus.cc:2894] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 08:14:16.412585 10097 raft_consensus.cc:2866] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:14:16.412680 10097 raft_consensus.cc:2258] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:16.414491 10097 ts_tablet_manager.cc:1875] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:16.421844 10097 ts_tablet_manager.cc:1888] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:16.422073 10097 log.cc:1196] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-4/wal/wals/cfc0c7b429754d92b4dd166c073ac8ed
I20240501 08:14:16.422598 10097 ts_tablet_manager.cc:902] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:16.447373 10036 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:16.463872 8826 raft_consensus.cc:1203] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:16.470882 10231 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:16.471473 10231 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:16.471786 10231 ts_tablet_manager.cc:1367] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.118s user 0.037s sys 0.019s
I20240501 08:14:16.473170 10231 raft_consensus.cc:348] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:16.474052 10231 raft_consensus.cc:729] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:16.474097 8826 raft_consensus.cc:1203] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:16.474397 10231 consensus_queue.cc:260] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:16.500523 6896 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:16.507424 6491 catalog_manager.cc:5484] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:16.542685 10243 tablet_copy_client.cc:308] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:14:16.543756 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet cfc0c7b429754d92b4dd166c073ac8ed from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:40161)
I20240501 08:14:16.543964 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet cfc0c7b429754d92b4dd166c073ac8ed from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:40161: session id = aba7c226aaca4e8c90931f547d48aaa3-cfc0c7b429754d92b4dd166c073ac8ed
I20240501 08:14:16.547237 6919 tablet_copy_source_session.cc:205] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:16.549189 10243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc0c7b429754d92b4dd166c073ac8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:16.552822 10271 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:16.565009 6637 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:16.576511 10073 ts_tablet_manager.cc:1398] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.297s user 0.051s sys 0.097s
I20240501 08:14:16.578383 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-13b4e2d5fe4846e8b3c6081abb3c53da received from {username='slave'} at 127.25.107.8:58105
I20240501 08:14:16.578560 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-13b4e2d5fe4846e8b3c6081abb3c53da on tablet 13b4e2d5fe4846e8b3c6081abb3c53da with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:16.592337 10243 tablet_copy_client.cc:783] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:16.592689 10243 tablet_copy_client.cc:647] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:16.594642 10243 tablet_copy_client.cc:523] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:16.598233 10243 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
W20240501 08:14:16.602061 6704 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 61777.5us
I20240501 08:14:16.604209 7026 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:16.611796 6495 catalog_manager.cc:5484] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:16.654605 6896 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:16.692420 10231 ts_tablet_manager.cc:1398] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.220s user 0.017s sys 0.045s
I20240501 08:14:16.693652 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-491674dfbf74498e8a787bebaa5e95da received from {username='slave'} at 127.25.107.8:58105
I20240501 08:14:16.693847 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-491674dfbf74498e8a787bebaa5e95da on tablet 491674dfbf74498e8a787bebaa5e95da with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:16.711707 8826 raft_consensus.cc:1203] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:16.715636 10097 tablet_copy_client.cc:308] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:14:16.718402 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet cfc0c7b429754d92b4dd166c073ac8ed from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:46197)
I20240501 08:14:16.718642 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet cfc0c7b429754d92b4dd166c073ac8ed from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:46197: session id = 0a814bba1abe4653bcd418a83ec42370-cfc0c7b429754d92b4dd166c073ac8ed
I20240501 08:14:16.721593 6919 tablet_copy_source_session.cc:205] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:16.723613 10097 ts_tablet_manager.cc:1875] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:14:16.734903 10097 ts_tablet_manager.cc:1888] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:14:16.735425 10097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc0c7b429754d92b4dd166c073ac8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:16.739455 10097 tablet_copy_client.cc:783] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:16.739778 10097 tablet_copy_client.cc:647] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:16.741814 10097 tablet_copy_client.cc:523] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:16.743182 10049 tablet_bootstrap.cc:492] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:16.743681 10049 tablet_bootstrap.cc:492] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:16.743981 10049 ts_tablet_manager.cc:1367] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.907s user 0.063s sys 0.005s
I20240501 08:14:16.744853 10049 raft_consensus.cc:348] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:16.745909 10097 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:16.780862 6896 consensus_queue.cc:237] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:16.793089 6768 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:16.794102 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 399d1a392b3f4d9e8531b6e0b5ad2a61. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:16.798306 10049 raft_consensus.cc:729] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:16.799744 10049 consensus_queue.cc:260] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:16.798337 8959 raft_consensus.cc:1203] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:16.813510 10329 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:16.817071 10049 ts_tablet_manager.cc:1398] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.073s user 0.004s sys 0.000s
I20240501 08:14:16.817948 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-4a6efa669c1645aaa971d59e5d38c8d5 received from {username='slave'} at 127.25.107.8:37833
I20240501 08:14:16.818107 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-4a6efa669c1645aaa971d59e5d38c8d5 on tablet 4a6efa669c1645aaa971d59e5d38c8d5 with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:16.818930 6637 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:16.819708 9757 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:16.836437 8959 raft_consensus.cc:1203] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:16.841213 8959 raft_consensus.cc:1203] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:16.848691 10328 ts_tablet_manager.cc:902] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:16.849617 10325 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:16.849973 6637 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:16.856431 10341 ts_tablet_manager.cc:902] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:16.863210 6768 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:16.867990 6497 catalog_manager.cc:5484] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:16.876616 10328 tablet_copy_client.cc:308] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:16.892576 10341 tablet_copy_client.cc:308] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:16.894695 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 60238162349647c681685349c6f8ede4 from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:46523)
I20240501 08:14:16.894922 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 60238162349647c681685349c6f8ede4 from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:46523: session id = aba7c226aaca4e8c90931f547d48aaa3-60238162349647c681685349c6f8ede4
I20240501 08:14:16.899374 7049 tablet_copy_source_session.cc:205] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:16.919631 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:16.944679 10243 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:16.945178 10243 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:16.945489 10243 ts_tablet_manager.cc:1367] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.347s user 0.045s sys 0.017s
I20240501 08:14:16.946386 10243 raft_consensus.cc:348] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:16.946661 10243 raft_consensus.cc:729] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:16.950722 10243 consensus_queue.cc:260] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:16.963101 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:16.964074 10243 ts_tablet_manager.cc:1398] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.018s user 0.006s sys 0.002s
I20240501 08:14:16.968454 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:16.990634 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-cfc0c7b429754d92b4dd166c073ac8ed received from {username='slave'} at 127.25.107.6:40161
I20240501 08:14:16.990789 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-cfc0c7b429754d92b4dd166c073ac8ed on tablet cfc0c7b429754d92b4dd166c073ac8ed with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:17.016877 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:17.042048 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:17.052564 6637 consensus_queue.cc:237] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:17.056267 6898 raft_consensus.cc:1261] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:17.056856 9872 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:17.059285 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:17.080701 10341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60238162349647c681685349c6f8ede4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:17.088627 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:17.089687 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:17.112111 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:17.117408 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:17.121428 6767 raft_consensus.cc:1261] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:17.134214 7028 consensus_queue.cc:237] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:17.139698 9872 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:17.141602 10341 tablet_copy_client.cc:783] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:17.141901 10341 tablet_copy_client.cc:647] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:17.163002 10341 tablet_copy_client.cc:523] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:17.174135 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:17.177551 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:17.185231 10341 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:17.210729 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:17.246095 9872 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:17.252314 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:17.252336 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:17.250075 6768 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:17.264178 6898 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:17.271078 6494 catalog_manager.cc:5484] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:14:17.280658 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9e70954aa4e4fcfbad12e8bfe3e0059. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:17.282373 6898 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:17.283003 9827 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:14:17.304378 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29506d751571401f86bced7e6f5415d7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:17.314465 10396 ts_tablet_manager.cc:902] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:17.319103 8695 raft_consensus.cc:1203] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 08:14:17.319869 6704 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 60865.5us
I20240501 08:14:17.323037 10097 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:17.324003 6768 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:17.327788 9827 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:17.331373 10341 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:17.331976 10341 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:17.332312 10341 ts_tablet_manager.cc:1367] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.147s user 0.047s sys 0.003s
I20240501 08:14:17.332736 9827 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:17.333156 10341 raft_consensus.cc:348] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:17.333375 10341 raft_consensus.cc:729] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:17.333655 10341 consensus_queue.cc:260] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:17.334921 10341 ts_tablet_manager.cc:1398] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:17.341179 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-60238162349647c681685349c6f8ede4 received from {username='slave'} at 127.25.107.6:46523
I20240501 08:14:17.341346 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-60238162349647c681685349c6f8ede4 on tablet 60238162349647c681685349c6f8ede4 with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:17.342221 8695 raft_consensus.cc:1203] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:17.343567 6898 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:17.345341 6488 catalog_manager.cc:5484] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:17.350337 6767 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:17.364895 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:17.386054 10097 tablet_bootstrap.cc:492] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:17.386572 10097 ts_tablet_manager.cc:1367] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.641s user 0.050s sys 0.012s
I20240501 08:14:17.387573 10097 raft_consensus.cc:348] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:17.387952 10097 raft_consensus.cc:729] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:17.388311 10097 consensus_queue.cc:260] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:17.393790 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet d7903106d7e74e1ea102b1abdbfce63e from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:55445)
I20240501 08:14:17.394201 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet d7903106d7e74e1ea102b1abdbfce63e from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:55445: session id = 0a814bba1abe4653bcd418a83ec42370-d7903106d7e74e1ea102b1abdbfce63e
I20240501 08:14:17.400091 10097 ts_tablet_manager.cc:1398] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240501 08:14:17.401172 6789 tablet_copy_source_session.cc:205] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:17.403506 10328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7903106d7e74e1ea102b1abdbfce63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:17.413437 10328 tablet_copy_client.cc:783] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:17.413831 10328 tablet_copy_client.cc:647] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:17.414484 10396 tablet_copy_client.cc:308] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:14:17.418180 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:46197)
I20240501 08:14:17.418350 6918 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-cfc0c7b429754d92b4dd166c073ac8ed received from {username='slave'} at 127.25.107.5:46197
I20240501 08:14:17.418473 6918 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-cfc0c7b429754d92b4dd166c073ac8ed on tablet cfc0c7b429754d92b4dd166c073ac8ed with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:17.419149 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:46197: session id = 0a814bba1abe4653bcd418a83ec42370-399d1a392b3f4d9e8531b6e0b5ad2a61
I20240501 08:14:17.422137 6919 tablet_copy_source_session.cc:205] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:17.425472 10328 tablet_copy_client.cc:523] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:17.436427 10328 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:17.438894 10396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:17.477656 10396 tablet_copy_client.cc:783] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:17.478863 10396 tablet_copy_client.cc:647] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:17.495553 10396 tablet_copy_client.cc:523] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:17.542021 10396 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:17.547998 8562 raft_consensus.cc:1203] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:17.564953 10341 ts_tablet_manager.cc:902] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:17.596395 10341 tablet_copy_client.cc:308] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:17.596628 6767 consensus_queue.cc:237] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:17.608119 6768 consensus_queue.cc:237] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:17.611577 6637 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:17.612216 7028 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:17.612900 10404 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:17.613946 10271 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:14:17.615810 6705 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a25e1ddb987b4f8c8a4031053f581c7c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:17.621968 7029 raft_consensus.cc:1261] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:17.622923 10271 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:17.632737 6638 raft_consensus.cc:1261] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:17.633459 10271 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:17.644776 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 29506d751571401f86bced7e6f5415d7 from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:33725)
I20240501 08:14:17.644999 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 29506d751571401f86bced7e6f5415d7 from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:33725: session id = aba7c226aaca4e8c90931f547d48aaa3-29506d751571401f86bced7e6f5415d7
I20240501 08:14:17.645587 10271 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:17.649194 7026 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:17.653230 6658 tablet_copy_source_session.cc:205] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:17.655346 10341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29506d751571401f86bced7e6f5415d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:14:17.656358 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdaad8e1d86942ddbc97958dcf31b344. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:17.657729 6488 catalog_manager.cc:5484] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:17.661626 10404 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:17.662997 6638 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:17.667598 7028 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:17.668309 10341 tablet_copy_client.cc:783] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:17.668637 10341 tablet_copy_client.cc:647] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:17.676752 10341 tablet_copy_client.cc:523] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:17.679502 6497 catalog_manager.cc:5484] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:17.704085 10341 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:17.708590 6638 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:17.768666 10462 ts_tablet_manager.cc:902] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:17.781299 10328 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:17.782090 10328 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:17.782459 10328 ts_tablet_manager.cc:1367] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.346s user 0.053s sys 0.003s
I20240501 08:14:17.782809 10396 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:17.783211 10396 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:17.786828 10328 raft_consensus.cc:348] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:17.787096 10328 raft_consensus.cc:729] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:17.789083 10396 ts_tablet_manager.cc:1367] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.247s user 0.046s sys 0.007s
I20240501 08:14:17.788913 10328 consensus_queue.cc:260] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:17.791357 10328 ts_tablet_manager.cc:1398] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:14:17.791520 10396 raft_consensus.cc:348] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:17.791751 10396 raft_consensus.cc:729] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:17.792805 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-d7903106d7e74e1ea102b1abdbfce63e received from {username='slave'} at 127.25.107.5:55445
I20240501 08:14:17.792990 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-d7903106d7e74e1ea102b1abdbfce63e on tablet d7903106d7e74e1ea102b1abdbfce63e with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:17.794940 10396 consensus_queue.cc:260] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:17.795687 10396 ts_tablet_manager.cc:1398] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:14:17.797446 10462 tablet_copy_client.cc:308] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:17.797462 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-399d1a392b3f4d9e8531b6e0b5ad2a61 received from {username='slave'} at 127.25.107.5:46197
I20240501 08:14:17.797649 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-399d1a392b3f4d9e8531b6e0b5ad2a61 on tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:17.798996 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet a9e70954aa4e4fcfbad12e8bfe3e0059 from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:37833)
I20240501 08:14:17.799177 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet a9e70954aa4e4fcfbad12e8bfe3e0059 from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:37833: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-a9e70954aa4e4fcfbad12e8bfe3e0059
I20240501 08:14:17.802174 7049 tablet_copy_source_session.cc:205] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:17.804211 10462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e70954aa4e4fcfbad12e8bfe3e0059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:17.819272 10462 tablet_copy_client.cc:783] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:17.820214 10462 tablet_copy_client.cc:647] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:17.829449 10462 tablet_copy_client.cc:523] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:17.831177 8562 raft_consensus.cc:1203] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:17.833925 10462 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:17.845016 6767 consensus_queue.cc:237] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:17.858713 10341 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:17.859311 10341 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
W20240501 08:14:17.859409 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f93c2d0f9c7947bb83a50c7fb8c3518a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:17.859656 10341 ts_tablet_manager.cc:1367] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.156s user 0.046s sys 0.012s
I20240501 08:14:17.861151 6896 raft_consensus.cc:1261] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:17.862416 10459 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:17.871081 7028 raft_consensus.cc:1261] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:17.872292 10459 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:17.887982 10341 raft_consensus.cc:348] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:17.888284 10341 raft_consensus.cc:729] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:17.889236 10341 consensus_queue.cc:260] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:17.891747 10341 ts_tablet_manager.cc:1398] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.032s user 0.004s sys 0.000s
I20240501 08:14:17.893383 10459 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:17.895104 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-29506d751571401f86bced7e6f5415d7 received from {username='slave'} at 127.25.107.6:33725
I20240501 08:14:17.895274 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-29506d751571401f86bced7e6f5415d7 on tablet 29506d751571401f86bced7e6f5415d7 with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:17.895716 7028 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:17.904808 6497 catalog_manager.cc:5484] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:17.910051 6896 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:17.988993 7028 consensus_queue.cc:237] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:18.001766 8695 raft_consensus.cc:1203] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:18.004524 10462 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:18.006551 10462 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:18.007735 10462 ts_tablet_manager.cc:1367] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.174s user 0.051s sys 0.005s
I20240501 08:14:18.009675 10462 raft_consensus.cc:348] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:18.010466 10462 raft_consensus.cc:729] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:18.010848 10462 consensus_queue.cc:260] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:18.018219 10462 ts_tablet_manager.cc:1398] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
W20240501 08:14:18.022862 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9629fbbb4554edc9ba341d42b51f252. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:18.023339 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-a9e70954aa4e4fcfbad12e8bfe3e0059 received from {username='slave'} at 127.25.107.8:37833
I20240501 08:14:18.023468 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-a9e70954aa4e4fcfbad12e8bfe3e0059 on tablet a9e70954aa4e4fcfbad12e8bfe3e0059 with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:18.025004 6896 raft_consensus.cc:1261] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:18.026198 10036 consensus_queue.cc:1035] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.034938 6638 raft_consensus.cc:1261] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:18.036702 10036 consensus_queue.cc:1035] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.079684 6638 consensus_queue.cc:237] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:18.107964 10552 ts_tablet_manager.cc:902] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:18.112062 10036 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.124994 6637 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.130470 7028 raft_consensus.cc:1261] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:18.131273 9872 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.132278 6898 raft_consensus.cc:1261] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:18.133786 6572 consensus_peers.cc:485] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 049090443eaa4b18ba90cb60af9a87ab. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:18.136701 10385 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.140758 8959 raft_consensus.cc:1203] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:18.145723 6497 catalog_manager.cc:5484] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.167125 6896 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.214955 9872 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.219597 7029 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.226310 10328 ts_tablet_manager.cc:902] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:18.226672 6494 catalog_manager.cc:5484] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.264040 6768 consensus_queue.cc:237] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:18.264699 10552 tablet_copy_client.cc:308] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:18.273003 8562 raft_consensus.cc:1203] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:18.277889 6896 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.279731 10584 ts_tablet_manager.cc:902] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:18.288363 6637 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:18.296330 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet fdaad8e1d86942ddbc97958dcf31b344 from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:49989)
I20240501 08:14:18.296587 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet fdaad8e1d86942ddbc97958dcf31b344 from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:49989: session id = a518ca5e9de24204bd028b0ce4a77d58-fdaad8e1d86942ddbc97958dcf31b344
I20240501 08:14:18.299803 6789 tablet_copy_source_session.cc:205] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:18.301594 10552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdaad8e1d86942ddbc97958dcf31b344. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:14:18.303740 6574 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30b4f075057e42d0bb3f8cdabdfbc643. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:18.305483 7029 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:18.306535 9872 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.310973 10552 tablet_copy_client.cc:783] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:18.311689 10552 tablet_copy_client.cc:647] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:18.314749 10552 tablet_copy_client.cc:523] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:18.315752 6767 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:18.316491 9872 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.331365 10577 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:18.332175 7028 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:18.333590 10552 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
W20240501 08:14:18.336644 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 193de3c11ae54408a64ae3007bf88920. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:18.337796 6769 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:18.339275 6494 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:18.341140 7028 raft_consensus.cc:1261] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:18.341989 10459 consensus_queue.cc:1035] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.344067 6896 raft_consensus.cc:1261] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:18.344739 10271 consensus_queue.cc:1035] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.353926 10328 tablet_copy_client.cc:308] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:18.355638 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet a25e1ddb987b4f8c8a4031053f581c7c from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:55445)
I20240501 08:14:18.355932 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet a25e1ddb987b4f8c8a4031053f581c7c from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:55445: session id = 0a814bba1abe4653bcd418a83ec42370-a25e1ddb987b4f8c8a4031053f581c7c
I20240501 08:14:18.359933 6789 tablet_copy_source_session.cc:205] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:18.362176 10328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25e1ddb987b4f8c8a4031053f581c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:18.370601 10271 raft_consensus.cc:2932] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:18.369861 10328 tablet_copy_client.cc:783] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:18.376463 10328 tablet_copy_client.cc:647] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:18.379560 6768 consensus_queue.cc:237] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:18.380489 10328 tablet_copy_client.cc:523] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:18.406394 10328 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:18.419505 10584 tablet_copy_client.cc:308] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
W20240501 08:14:18.421974 6705 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:18.423338 7028 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:18.425343 6637 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:18.426113 10608 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:18.430027 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet f93c2d0f9c7947bb83a50c7fb8c3518a from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:49989)
I20240501 08:14:18.430331 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet f93c2d0f9c7947bb83a50c7fb8c3518a from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:49989: session id = a518ca5e9de24204bd028b0ce4a77d58-f93c2d0f9c7947bb83a50c7fb8c3518a
I20240501 08:14:18.433328 6789 tablet_copy_source_session.cc:205] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:18.435599 10584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93c2d0f9c7947bb83a50c7fb8c3518a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:18.439590 10271 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240501 08:14:18.442214 7029 raft_consensus.cc:2932] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:18.449425 10608 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:18.450711 6497 catalog_manager.cc:5484] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:18.452831 7029 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:18.460942 6497 catalog_manager.cc:5484] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:18.479140 6637 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:18.490046 6896 raft_consensus.cc:2932] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
W20240501 08:14:18.505035 6703 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a25e1ddb987b4f8c8a4031053f581c7c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:18.509378 10584 tablet_copy_client.cc:783] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:18.509903 10584 tablet_copy_client.cc:647] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:18.528889 10584 tablet_copy_client.cc:523] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:18.540903 10584 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:18.542866 10552 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:18.543349 10552 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:18.543682 10552 ts_tablet_manager.cc:1367] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.210s user 0.048s sys 0.012s
I20240501 08:14:18.544570 10552 raft_consensus.cc:348] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:18.544837 10552 raft_consensus.cc:729] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:18.545099 10552 consensus_queue.cc:260] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:18.552572 10552 ts_tablet_manager.cc:1398] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20240501 08:14:18.559433 7029 consensus_queue.cc:237] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:18.559764 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-fdaad8e1d86942ddbc97958dcf31b344 received from {username='slave'} at 127.25.107.7:49989
I20240501 08:14:18.559897 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-fdaad8e1d86942ddbc97958dcf31b344 on tablet fdaad8e1d86942ddbc97958dcf31b344 with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:18.563323 8826 raft_consensus.cc:1203] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 08:14:18.593420 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9e70954aa4e4fcfbad12e8bfe3e0059. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:18.595134 8959 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:18.596083 10036 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:18.601168 6896 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:18.601774 6768 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:18.602874 10036 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:18.603263 9827 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:18.622006 10661 ts_tablet_manager.cc:902] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:18.623288 10328 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:18.623804 10328 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:18.624114 10328 ts_tablet_manager.cc:1367] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.218s user 0.043s sys 0.007s
I20240501 08:14:18.624959 10328 raft_consensus.cc:348] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:18.625191 10328 raft_consensus.cc:729] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:18.625468 10328 consensus_queue.cc:260] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:18.630604 10328 ts_tablet_manager.cc:1398] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:14:18.631490 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-a25e1ddb987b4f8c8a4031053f581c7c received from {username='slave'} at 127.25.107.5:55445
I20240501 08:14:18.631640 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-a25e1ddb987b4f8c8a4031053f581c7c on tablet a25e1ddb987b4f8c8a4031053f581c7c with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:18.642983 10036 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:18.658986 6896 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:18.668836 6497 catalog_manager.cc:5484] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:18.684420 6768 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:18.687975 10661 tablet_copy_client.cc:308] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:18.688972 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet e9629fbbb4554edc9ba341d42b51f252 from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:37833)
I20240501 08:14:18.689157 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet e9629fbbb4554edc9ba341d42b51f252 from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:37833: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-e9629fbbb4554edc9ba341d42b51f252
I20240501 08:14:18.691992 7049 tablet_copy_source_session.cc:205] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:18.693917 10661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9629fbbb4554edc9ba341d42b51f252. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:18.693783 8959 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:18.720346 10661 tablet_copy_client.cc:783] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:18.724184 10661 tablet_copy_client.cc:647] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:18.726406 10661 tablet_copy_client.cc:523] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:18.732187 10661 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:18.741209 10689 ts_tablet_manager.cc:902] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:18.756239 10584 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:18.756775 10584 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:18.757104 10584 ts_tablet_manager.cc:1367] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.216s user 0.052s sys 0.010s
I20240501 08:14:18.757786 10584 raft_consensus.cc:348] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:18.758018 10584 raft_consensus.cc:729] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:18.758296 10584 consensus_queue.cc:260] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:18.758852 10584 ts_tablet_manager.cc:1398] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:18.759790 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-f93c2d0f9c7947bb83a50c7fb8c3518a received from {username='slave'} at 127.25.107.7:49989
I20240501 08:14:18.759938 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-f93c2d0f9c7947bb83a50c7fb8c3518a on tablet f93c2d0f9c7947bb83a50c7fb8c3518a with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:18.773133 10689 tablet_copy_client.cc:308] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:18.774070 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 049090443eaa4b18ba90cb60af9a87ab from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:58105)
I20240501 08:14:18.774281 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 049090443eaa4b18ba90cb60af9a87ab from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:58105: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-049090443eaa4b18ba90cb60af9a87ab
I20240501 08:14:18.775043 6768 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:18.777236 6658 tablet_copy_source_session.cc:205] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:18.778283 6896 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:18.778414 7029 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:18.778978 10689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049090443eaa4b18ba90cb60af9a87ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:18.779970 10608 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.782599 10606 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.787397 10689 tablet_copy_client.cc:783] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:18.787577 8826 raft_consensus.cc:1203] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:18.789330 10689 tablet_copy_client.cc:647] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:18.791813 10689 tablet_copy_client.cc:523] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 08:14:18.792963 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a56c284444496b8579600a49f19d1d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:18.794951 10689 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:18.818349 6637 consensus_queue.cc:237] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:18.822124 6898 raft_consensus.cc:1261] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:18.823654 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78d1df7b0ea44f89b9f22c156f907c1e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:18.823959 7028 raft_consensus.cc:1261] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:18.824627 9872 consensus_queue.cc:1035] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.825263 9872 consensus_queue.cc:1035] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:18.841995 8562 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:18.850899 10328 ts_tablet_manager.cc:902] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:18.853201 10608 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.857753 6899 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.861120 7029 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.861589 10584 ts_tablet_manager.cc:902] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:18.864650 9872 raft_consensus.cc:2932] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:18.865092 6494 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index -1 to 4, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:18.871076 7028 raft_consensus.cc:2932] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:18.875535 6497 catalog_manager.cc:5484] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:18.886440 6899 raft_consensus.cc:2932] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:18.914832 10713 ts_tablet_manager.cc:902] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:18.922756 10584 tablet_copy_client.cc:308] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:18.923473 10661 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:18.923661 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet 193de3c11ae54408a64ae3007bf88920 from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:49989)
I20240501 08:14:18.923838 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet 193de3c11ae54408a64ae3007bf88920 from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:49989: session id = a518ca5e9de24204bd028b0ce4a77d58-193de3c11ae54408a64ae3007bf88920
I20240501 08:14:18.924006 10661 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:18.924315 10661 ts_tablet_manager.cc:1367] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.192s user 0.041s sys 0.010s
I20240501 08:14:18.925210 10661 raft_consensus.cc:348] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:18.925508 10661 raft_consensus.cc:729] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:18.927430 6789 tablet_copy_source_session.cc:205] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:18.929649 10584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193de3c11ae54408a64ae3007bf88920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:18.930425 10661 consensus_queue.cc:260] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:18.934581 10661 ts_tablet_manager.cc:1398] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:14:18.940896 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-e9629fbbb4554edc9ba341d42b51f252 received from {username='slave'} at 127.25.107.8:37833
I20240501 08:14:18.941083 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-e9629fbbb4554edc9ba341d42b51f252 on tablet e9629fbbb4554edc9ba341d42b51f252 with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:18.953845 10689 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:18.954476 10689 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:18.954815 10689 ts_tablet_manager.cc:1367] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.160s user 0.044s sys 0.011s
I20240501 08:14:18.955711 10689 raft_consensus.cc:348] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:18.956005 10689 raft_consensus.cc:729] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:18.956133 10584 tablet_copy_client.cc:783] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:18.956416 10584 tablet_copy_client.cc:647] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:18.956305 10689 consensus_queue.cc:260] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:18.957697 10689 ts_tablet_manager.cc:1398] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:14:18.958508 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-049090443eaa4b18ba90cb60af9a87ab received from {username='slave'} at 127.25.107.8:58105
I20240501 08:14:18.958712 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-049090443eaa4b18ba90cb60af9a87ab on tablet 049090443eaa4b18ba90cb60af9a87ab with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:18.958839 10584 tablet_copy_client.cc:523] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:18.965579 10584 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:19.037689 10328 tablet_copy_client.cc:308] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:19.038568 10713 tablet_copy_client.cc:308] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:19.038667 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:51915)
I20240501 08:14:19.038858 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:51915: session id = 0a814bba1abe4653bcd418a83ec42370-30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:14:19.042171 6658 tablet_copy_source_session.cc:205] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:19.067445 10584 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:19.067962 10584 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:19.068238 10584 ts_tablet_manager.cc:1367] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.103s user 0.050s sys 0.006s
I20240501 08:14:19.069059 10584 raft_consensus.cc:348] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:19.069284 10584 raft_consensus.cc:729] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:19.069555 10584 consensus_queue.cc:260] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:19.071143 10584 ts_tablet_manager.cc:1398] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:14:19.071830 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-193de3c11ae54408a64ae3007bf88920 received from {username='slave'} at 127.25.107.7:49989
I20240501 08:14:19.071974 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-193de3c11ae54408a64ae3007bf88920 on tablet 193de3c11ae54408a64ae3007bf88920 with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:19.075070 10328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4f075057e42d0bb3f8cdabdfbc643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:19.088182 8959 raft_consensus.cc:1203] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:19.094435 10328 tablet_copy_client.cc:783] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:19.094854 10328 tablet_copy_client.cc:647] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:19.096876 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet a25e1ddb987b4f8c8a4031053f581c7c from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:52933)
I20240501 08:14:19.097091 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet a25e1ddb987b4f8c8a4031053f581c7c from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:52933: session id = aba7c226aaca4e8c90931f547d48aaa3-a25e1ddb987b4f8c8a4031053f581c7c
I20240501 08:14:19.097244 7028 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:19.099658 10328 tablet_copy_client.cc:523] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:19.100287 6789 tablet_copy_source_session.cc:205] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:19.101856 10713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25e1ddb987b4f8c8a4031053f581c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:19.103682 10328 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:19.110442 10713 tablet_copy_client.cc:783] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:19.110914 10713 tablet_copy_client.cc:647] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:19.117830 10713 tablet_copy_client.cc:523] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:19.132853 8959 raft_consensus.cc:1203] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 08:14:19.138489 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f090fda814f455a88dccd6680bdd05f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:19.141240 10770 ts_tablet_manager.cc:902] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:19.142731 8563 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:19.145139 6637 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:19.146384 10036 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:19.147099 10036 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:19.154731 10036 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:19.159205 6899 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:19.159325 8562 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:19.160295 9827 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:19.167305 6494 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 reported cstate change: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:19.170490 10770 tablet_copy_client.cc:308] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:19.171388 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet a9e70954aa4e4fcfbad12e8bfe3e0059 from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:34223)
I20240501 08:14:19.171586 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet a9e70954aa4e4fcfbad12e8bfe3e0059 from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:34223: session id = 0a814bba1abe4653bcd418a83ec42370-a9e70954aa4e4fcfbad12e8bfe3e0059
I20240501 08:14:19.173938 7049 tablet_copy_source_session.cc:205] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:19.175508 10770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e70954aa4e4fcfbad12e8bfe3e0059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:19.175155 6637 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:19.186939 10770 tablet_copy_client.cc:783] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:19.187386 10770 tablet_copy_client.cc:647] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:19.189287 10770 tablet_copy_client.cc:523] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:19.193241 10770 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:19.195750 10713 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:19.201606 6899 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:19.233485 10689 ts_tablet_manager.cc:902] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:19.249295 10584 ts_tablet_manager.cc:902] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:19.257629 10770 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:19.258247 10770 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:19.258674 10770 ts_tablet_manager.cc:1367] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.066s user 0.047s sys 0.009s
I20240501 08:14:19.259678 10770 raft_consensus.cc:348] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:19.260068 10770 raft_consensus.cc:729] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:19.260490 10770 consensus_queue.cc:260] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:19.261374 10770 ts_tablet_manager.cc:1398] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:19.262784 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-a9e70954aa4e4fcfbad12e8bfe3e0059 received from {username='slave'} at 127.25.107.5:34223
I20240501 08:14:19.262915 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-a9e70954aa4e4fcfbad12e8bfe3e0059 on tablet a9e70954aa4e4fcfbad12e8bfe3e0059 with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:19.281385 6637 consensus_queue.cc:237] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:19.297070 10584 tablet_copy_client.cc:308] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:19.297883 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 78d1df7b0ea44f89b9f22c156f907c1e from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:48573)
I20240501 08:14:19.298112 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 78d1df7b0ea44f89b9f22c156f907c1e from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:48573: session id = a518ca5e9de24204bd028b0ce4a77d58-78d1df7b0ea44f89b9f22c156f907c1e
I20240501 08:14:19.301474 6658 tablet_copy_source_session.cc:205] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 08:14:19.302578 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13b4e2d5fe4846e8b3c6081abb3c53da. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:19.304363 10584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d1df7b0ea44f89b9f22c156f907c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:19.305413 8959 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:19.306675 10689 tablet_copy_client.cc:308] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:19.306844 9872 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:19.313163 10584 tablet_copy_client.cc:783] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:19.313458 10584 tablet_copy_client.cc:647] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:19.314844 8826 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:19.316147 9872 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:19.317284 10584 tablet_copy_client.cc:523] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:19.317719 6899 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:19.321863 10708 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:19.323652 10584 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:19.348692 6768 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:19.350422 10708 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:19.351120 10328 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:19.351691 10328 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:19.352064 10328 ts_tablet_manager.cc:1367] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.248s user 0.041s sys 0.010s
I20240501 08:14:19.353019 10328 raft_consensus.cc:348] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:19.353375 10328 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:19.353682 10328 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:19.357960 10328 ts_tablet_manager.cc:1398] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 08:14:19.360846 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-30b4f075057e42d0bb3f8cdabdfbc643 received from {username='slave'} at 127.25.107.5:51915
I20240501 08:14:19.361104 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-30b4f075057e42d0bb3f8cdabdfbc643 on tablet 30b4f075057e42d0bb3f8cdabdfbc643 with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:19.377207 9872 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:19.381378 6898 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:19.392172 6497 catalog_manager.cc:5484] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:19.395372 8826 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:19.404465 8959 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:19.432435 8826 raft_consensus.cc:1203] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:19.437633 6769 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:19.470868 6768 consensus_queue.cc:237] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:19.477576 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet a7a56c284444496b8579600a49f19d1d from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:41771)
I20240501 08:14:19.477788 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet a7a56c284444496b8579600a49f19d1d from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:41771: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-a7a56c284444496b8579600a49f19d1d
I20240501 08:14:19.480739 6789 tablet_copy_source_session.cc:205] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:19.484220 10689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a56c284444496b8579600a49f19d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:19.491647 10689 tablet_copy_client.cc:783] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:19.491925 10689 tablet_copy_client.cc:647] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:19.491935 6898 raft_consensus.cc:1261] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:19.492506 6703 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 358b6fe9c68349cb8cb963bb3a9ed2e9. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:19.493274 10606 consensus_queue.cc:1035] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:19.495064 6637 raft_consensus.cc:1261] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:19.495684 10606 consensus_queue.cc:1035] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:19.496763 10689 tablet_copy_client.cc:523] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:19.500571 10689 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:19.538861 10606 raft_consensus.cc:2932] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:19.539616 6899 raft_consensus.cc:2932] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:19.546407 6488 catalog_manager.cc:5484] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:19.546449 6638 raft_consensus.cc:2932] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:19.566499 10713 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:19.567173 10713 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:19.567545 10713 ts_tablet_manager.cc:1367] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.372s user 0.040s sys 0.058s
I20240501 08:14:19.568472 10713 raft_consensus.cc:348] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:19.568846 10713 raft_consensus.cc:729] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:19.569200 10713 consensus_queue.cc:260] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:19.582005 10713 ts_tablet_manager.cc:1398] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20240501 08:14:19.582875 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-a25e1ddb987b4f8c8a4031053f581c7c received from {username='slave'} at 127.25.107.6:52933
I20240501 08:14:19.583053 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-a25e1ddb987b4f8c8a4031053f581c7c on tablet a25e1ddb987b4f8c8a4031053f581c7c with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:19.642832 10856 ts_tablet_manager.cc:902] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:19.647207 8563 raft_consensus.cc:1203] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:19.670876 10584 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:19.671507 10584 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:19.671908 10584 ts_tablet_manager.cc:1367] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.348s user 0.081s sys 0.041s
I20240501 08:14:19.672833 10584 raft_consensus.cc:348] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:19.673178 10584 raft_consensus.cc:729] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:19.673504 10584 consensus_queue.cc:260] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:19.674306 10584 ts_tablet_manager.cc:1398] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:19.675383 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-78d1df7b0ea44f89b9f22c156f907c1e received from {username='slave'} at 127.25.107.7:48573
I20240501 08:14:19.675578 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-78d1df7b0ea44f89b9f22c156f907c1e on tablet 78d1df7b0ea44f89b9f22c156f907c1e with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:19.675941 10713 ts_tablet_manager.cc:902] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:19.679477 8563 raft_consensus.cc:1203] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:19.686754 6638 consensus_queue.cc:237] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
W20240501 08:14:19.704816 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4db7f887af8f425c8fa661196add4d37. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:19.707473 6768 raft_consensus.cc:1261] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:19.708308 10708 consensus_queue.cc:1035] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:19.715852 6898 raft_consensus.cc:1261] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:19.716698 10708 consensus_queue.cc:1035] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:19.722445 10856 tablet_copy_client.cc:308] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:19.724009 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 5f090fda814f455a88dccd6680bdd05f from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:37833)
I20240501 08:14:19.724215 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 5f090fda814f455a88dccd6680bdd05f from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:37833: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-5f090fda814f455a88dccd6680bdd05f
I20240501 08:14:19.727178 8695 raft_consensus.cc:1203] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:19.727864 7049 tablet_copy_source_session.cc:205] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:19.729378 10856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f090fda814f455a88dccd6680bdd05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:19.736864 10708 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:19.747316 6899 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:19.754870 6768 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:19.755321 6488 catalog_manager.cc:5484] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:19.775190 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:19.791249 6768 consensus_queue.cc:237] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:19.796317 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:19.809679 8695 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:14:19.809852 6702 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a25e1ddb987b4f8c8a4031053f581c7c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:19.811264 7028 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:19.811285 8563 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:19.811952 10608 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:19.812240 10839 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:19.812295 10856 tablet_copy_client.cc:783] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:19.812630 10856 tablet_copy_client.cc:647] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:19.813400 6638 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:19.814651 10606 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:19.815158 10606 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:19.816728 10856 tablet_copy_client.cc:523] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:19.828011 10856 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:19.855015 10689 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:19.855546 10689 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:19.866730 10713 tablet_copy_client.cc:308] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:19.867745 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 13b4e2d5fe4846e8b3c6081abb3c53da from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:33725)
I20240501 08:14:19.867988 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 13b4e2d5fe4846e8b3c6081abb3c53da from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:33725: session id = aba7c226aaca4e8c90931f547d48aaa3-13b4e2d5fe4846e8b3c6081abb3c53da
I20240501 08:14:19.875470 6658 tablet_copy_source_session.cc:205] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:19.882139 10713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b4e2d5fe4846e8b3c6081abb3c53da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:19.883925 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:19.896703 10713 tablet_copy_client.cc:783] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:19.897064 10713 tablet_copy_client.cc:647] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:19.903419 10713 tablet_copy_client.cc:523] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:19.909955 10713 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:19.912375 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:19.913640 10689 ts_tablet_manager.cc:1367] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.413s user 0.053s sys 0.013s
I20240501 08:14:19.914906 10689 raft_consensus.cc:348] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:19.915252 10689 raft_consensus.cc:729] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:19.915581 10689 consensus_queue.cc:260] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:19.917177 10608 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:19.920725 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:19.923115 7027 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:19.925419 8563 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:19.932824 6497 catalog_manager.cc:5484] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 reported cstate change: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:19.936190 8695 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:19.964124 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:19.952548 6638 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:19.987291 10689 ts_tablet_manager.cc:1398] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.074s user 0.002s sys 0.000s
I20240501 08:14:19.992512 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-a7a56c284444496b8579600a49f19d1d received from {username='slave'} at 127.25.107.8:41771
I20240501 08:14:19.992698 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-a7a56c284444496b8579600a49f19d1d on tablet a7a56c284444496b8579600a49f19d1d with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:20.011106 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:20.037889 6768 consensus_queue.cc:237] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:20.041607 10713 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:20.045295 10713 tablet_bootstrap.cc:492] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:20.045699 10713 ts_tablet_manager.cc:1367] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.136s user 0.041s sys 0.013s
I20240501 08:14:20.047781 10713 raft_consensus.cc:348] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:20.048066 10713 raft_consensus.cc:729] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:20.048321 10713 consensus_queue.cc:260] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:20.048945 8959 raft_consensus.cc:1203] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 08:14:20.055318 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f93c2d0f9c7947bb83a50c7fb8c3518a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:20.056432 8826 raft_consensus.cc:1261] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:20.057070 10608 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:20.060851 10713 ts_tablet_manager.cc:1398] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 08:14:20.062714 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-13b4e2d5fe4846e8b3c6081abb3c53da received from {username='slave'} at 127.25.107.6:33725
I20240501 08:14:20.062876 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-13b4e2d5fe4846e8b3c6081abb3c53da on tablet 13b4e2d5fe4846e8b3c6081abb3c53da with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:20.063922 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:20.065685 8695 raft_consensus.cc:1203] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:14:20.074388 7028 raft_consensus.cc:1261] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:20.075079 10839 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:20.077527 6898 raft_consensus.cc:1261] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:20.078413 10839 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:20.106565 10608 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:20.109943 6899 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:20.117978 10713 ts_tablet_manager.cc:902] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:20.119829 6488 catalog_manager.cc:5484] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:20.124694 8826 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:20.126940 7028 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:20.180581 8826 raft_consensus.cc:1203] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:20.198523 6638 consensus_queue.cc:237] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:20.211158 8695 raft_consensus.cc:1261] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:14:20.211248 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29506d751571401f86bced7e6f5415d7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:20.211751 9872 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:20.212280 6769 raft_consensus.cc:1261] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:20.212339 6898 raft_consensus.cc:1261] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:20.213148 9872 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:20.215145 9872 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:20.232578 9872 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:20.236676 8695 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:20.237653 6769 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:20.238126 6899 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:20.243911 6494 catalog_manager.cc:5484] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:20.263690 10984 ts_tablet_manager.cc:902] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:20.268510 10856 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:20.269074 10856 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:20.269394 10856 ts_tablet_manager.cc:1367] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.442s user 0.054s sys 0.000s
I20240501 08:14:20.270318 10856 raft_consensus.cc:348] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:20.270610 10856 raft_consensus.cc:729] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:20.270851 10856 consensus_queue.cc:260] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:20.271472 10856 ts_tablet_manager.cc:1398] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:20.272876 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-5f090fda814f455a88dccd6680bdd05f received from {username='slave'} at 127.25.107.8:37833
I20240501 08:14:20.273037 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-5f090fda814f455a88dccd6680bdd05f on tablet 5f090fda814f455a88dccd6680bdd05f with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:20.289891 10713 tablet_copy_client.cc:308] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:20.290696 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:52933)
I20240501 08:14:20.290899 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:52933: session id = aba7c226aaca4e8c90931f547d48aaa3-358b6fe9c68349cb8cb963bb3a9ed2e9
I20240501 08:14:20.293923 6789 tablet_copy_source_session.cc:205] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:20.297374 10713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:20.299912 10995 ts_tablet_manager.cc:902] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:20.309033 10995 tablet_copy_client.cc:308] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:20.311394 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet a25e1ddb987b4f8c8a4031053f581c7c from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:49989)
I20240501 08:14:20.311610 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet a25e1ddb987b4f8c8a4031053f581c7c from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:49989: session id = a518ca5e9de24204bd028b0ce4a77d58-a25e1ddb987b4f8c8a4031053f581c7c
I20240501 08:14:20.314996 6789 tablet_copy_source_session.cc:205] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:20.319979 10995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25e1ddb987b4f8c8a4031053f581c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:20.326092 10984 tablet_copy_client.cc:308] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:20.327381 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 4db7f887af8f425c8fa661196add4d37 from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:51915)
I20240501 08:14:20.327544 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 4db7f887af8f425c8fa661196add4d37 from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:51915: session id = 0a814bba1abe4653bcd418a83ec42370-4db7f887af8f425c8fa661196add4d37
I20240501 08:14:20.330742 6658 tablet_copy_source_session.cc:205] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:20.332302 10984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db7f887af8f425c8fa661196add4d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:20.342583 10995 tablet_copy_client.cc:783] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:20.342865 10995 tablet_copy_client.cc:647] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:20.344036 10984 tablet_copy_client.cc:783] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:20.344372 10984 tablet_copy_client.cc:647] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:20.344666 10995 tablet_copy_client.cc:523] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:20.346220 10984 tablet_copy_client.cc:523] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:20.349814 10984 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:20.349917 10995 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:20.359010 10713 tablet_copy_client.cc:783] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:20.359302 10713 tablet_copy_client.cc:647] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:20.374197 10713 tablet_copy_client.cc:523] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:20.391291 10713 tablet_bootstrap.cc:492] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:20.410696 10984 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:20.421504 6899 consensus_queue.cc:237] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:20.436259 8959 raft_consensus.cc:1203] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:20.442607 10984 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:20.442991 10984 ts_tablet_manager.cc:1367] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.093s user 0.052s sys 0.000s
I20240501 08:14:20.443889 10984 raft_consensus.cc:348] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:20.444173 10984 raft_consensus.cc:729] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:20.444413 10984 consensus_queue.cc:260] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:20.468279 8563 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:20.469038 10671 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:20.479182 6638 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:20.482467 10671 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:20.488368 6769 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:20.489060 10671 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:20.510727 10984 ts_tablet_manager.cc:1398] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.068s user 0.004s sys 0.012s
I20240501 08:14:20.512631 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-4db7f887af8f425c8fa661196add4d37 received from {username='slave'} at 127.25.107.5:51915
I20240501 08:14:20.512792 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-4db7f887af8f425c8fa661196add4d37 on tablet 4db7f887af8f425c8fa661196add4d37 with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:20.521574 6769 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:20.520385 11045 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:20.528815 6488 catalog_manager.cc:5484] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:20.558837 6638 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:14:20.591611 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 399d1a392b3f4d9e8531b6e0b5ad2a61. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:20.597715 6835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 62792us
I20240501 08:14:20.605964 8563 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:20.616173 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:20.666379 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:20.668711 10984 ts_tablet_manager.cc:902] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:20.676182 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:20.703529 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:20.707195 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:20.713310 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:20.715571 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:20.742712 11078 ts_tablet_manager.cc:902] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:20.749495 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:20.779749 10984 tablet_copy_client.cc:308] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:20.780612 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet f93c2d0f9c7947bb83a50c7fb8c3518a from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:55445)
I20240501 08:14:20.780803 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet f93c2d0f9c7947bb83a50c7fb8c3518a from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:55445: session id = 0a814bba1abe4653bcd418a83ec42370-f93c2d0f9c7947bb83a50c7fb8c3518a
I20240501 08:14:20.783727 6789 tablet_copy_source_session.cc:205] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:20.806648 8563 raft_consensus.cc:1203] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:20.813485 10995 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:20.814019 10995 tablet_bootstrap.cc:492] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:20.821405 10995 ts_tablet_manager.cc:1367] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.471s user 0.040s sys 0.012s
I20240501 08:14:20.822494 10995 raft_consensus.cc:348] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:20.822822 10995 raft_consensus.cc:729] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:20.823096 10995 consensus_queue.cc:260] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:20.830479 10995 ts_tablet_manager.cc:1398] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20240501 08:14:20.831739 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-a25e1ddb987b4f8c8a4031053f581c7c received from {username='slave'} at 127.25.107.7:49989
I20240501 08:14:20.831943 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-a25e1ddb987b4f8c8a4031053f581c7c on tablet a25e1ddb987b4f8c8a4031053f581c7c with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:20.832412 10984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f93c2d0f9c7947bb83a50c7fb8c3518a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:20.845139 8826 raft_consensus.cc:1203] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:14:20.860628 10984 tablet_copy_client.cc:783] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:20.860953 10984 tablet_copy_client.cc:647] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:20.862819 10984 tablet_copy_client.cc:523] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:20.897671 10984 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:20.903126 11078 tablet_copy_client.cc:308] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:20.904112 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 29506d751571401f86bced7e6f5415d7 from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:48573)
I20240501 08:14:20.904335 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 29506d751571401f86bced7e6f5415d7 from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:48573: session id = a518ca5e9de24204bd028b0ce4a77d58-29506d751571401f86bced7e6f5415d7
I20240501 08:14:20.908391 6658 tablet_copy_source_session.cc:205] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:20.910887 11078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29506d751571401f86bced7e6f5415d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:20.932196 11078 tablet_copy_client.cc:783] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:20.938628 11078 tablet_copy_client.cc:647] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:20.941413 11078 tablet_copy_client.cc:523] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:20.964706 11078 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:20.975037 11112 ts_tablet_manager.cc:902] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:21.048338 10984 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:21.049077 10984 tablet_bootstrap.cc:492] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:21.049584 10984 ts_tablet_manager.cc:1367] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.152s user 0.040s sys 0.014s
I20240501 08:14:21.050603 10984 raft_consensus.cc:348] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:21.051090 10984 raft_consensus.cc:729] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:21.051470 10984 consensus_queue.cc:260] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:21.064075 10984 ts_tablet_manager.cc:1398] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 08:14:21.074039 10713 tablet_bootstrap.cc:492] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:21.074180 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-f93c2d0f9c7947bb83a50c7fb8c3518a received from {username='slave'} at 127.25.107.5:55445
I20240501 08:14:21.074374 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-f93c2d0f9c7947bb83a50c7fb8c3518a on tablet f93c2d0f9c7947bb83a50c7fb8c3518a with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:21.079375 10713 tablet_bootstrap.cc:492] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:21.079892 10713 ts_tablet_manager.cc:1367] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.689s user 0.050s sys 0.004s
I20240501 08:14:21.080962 10713 raft_consensus.cc:348] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:21.081458 10713 raft_consensus.cc:729] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:21.081791 10713 consensus_queue.cc:260] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:21.089450 10713 ts_tablet_manager.cc:1398] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.009s user 0.001s sys 0.007s
I20240501 08:14:21.090561 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-358b6fe9c68349cb8cb963bb3a9ed2e9 received from {username='slave'} at 127.25.107.6:52933
I20240501 08:14:21.090723 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-358b6fe9c68349cb8cb963bb3a9ed2e9 on tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:21.112154 6638 consensus_queue.cc:237] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:21.126053 6899 raft_consensus.cc:1261] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:21.126787 10577 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:21.127291 7028 raft_consensus.cc:1261] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:21.127753 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b9bd7500e8747c6b8ee94a1faa1c5f7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:21.128405 10577 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:21.142213 10577 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:21.144735 7027 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:21.173537 8563 raft_consensus.cc:1203] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:21.183840 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:21.186055 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:21.217738 6494 catalog_manager.cc:5484] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
W20240501 08:14:21.221325 6573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 58470.2us
W20240501 08:14:21.222844 6571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 60131.8us
I20240501 08:14:21.224112 6899 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:21.235321 11112 tablet_copy_client.cc:308] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:14:21.245746 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:21.258769 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:21.275310 8695 raft_consensus.cc:1203] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:21.302431 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:21.310125 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:21.318521 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:21.329905 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:32865)
I20240501 08:14:21.330102 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:32865: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-399d1a392b3f4d9e8531b6e0b5ad2a61
I20240501 08:14:21.333524 6919 tablet_copy_source_session.cc:205] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:21.340268 11112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:21.347815 7028 consensus_queue.cc:237] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:21.350203 11112 tablet_copy_client.cc:783] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:21.350558 11112 tablet_copy_client.cc:647] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
W20240501 08:14:21.350914 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9e70954aa4e4fcfbad12e8bfe3e0059. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:21.352041 8563 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:21.352674 10974 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:21.354622 11112 tablet_copy_client.cc:523] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:21.355036 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:21.356194 6899 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:21.356818 10974 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:21.360844 6769 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:21.361469 10974 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:21.363032 11112 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:21.403343 8959 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:21.404362 10974 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:21.441743 10971 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:21.450879 8959 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:21.456094 6494 catalog_manager.cc:5484] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:21.462188 6899 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:21.464797 8563 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:21.488226 6769 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:21.489863 11112 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:21.522598 11078 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:21.523156 11078 tablet_bootstrap.cc:492] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:21.523479 11078 ts_tablet_manager.cc:1367] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.559s user 0.046s sys 0.007s
I20240501 08:14:21.524854 11112 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:21.525238 11078 raft_consensus.cc:348] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:21.525566 11078 raft_consensus.cc:729] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:21.525846 11078 consensus_queue.cc:260] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:21.527135 11112 ts_tablet_manager.cc:1367] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.164s user 0.054s sys 0.006s
I20240501 08:14:21.529920 11112 raft_consensus.cc:348] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:21.532307 11112 raft_consensus.cc:729] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:21.532697 11112 consensus_queue.cc:260] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:21.536691 11112 ts_tablet_manager.cc:1398] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 08:14:21.538442 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-399d1a392b3f4d9e8531b6e0b5ad2a61 received from {username='slave'} at 127.25.107.8:32865
I20240501 08:14:21.538604 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-399d1a392b3f4d9e8531b6e0b5ad2a61 on tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:21.576398 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:21.588719 11078 ts_tablet_manager.cc:1398] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.065s user 0.004s sys 0.006s
I20240501 08:14:21.594293 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-29506d751571401f86bced7e6f5415d7 received from {username='slave'} at 127.25.107.7:48573
I20240501 08:14:21.594453 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-29506d751571401f86bced7e6f5415d7 on tablet 29506d751571401f86bced7e6f5415d7 with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:21.601969 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:21.644210 7028 consensus_queue.cc:237] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:21.650770 6769 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:21.651466 6963 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45d076303ea24d7fa0fed5a9d132019c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:21.651818 10971 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:21.655831 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:21.655855 6638 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:21.656458 10971 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:21.708552 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:21.714879 11218 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:21.715798 6769 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:21.727106 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:21.728849 6494 catalog_manager.cc:5484] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:21.741669 11219 ts_tablet_manager.cc:902] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:21.777813 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:21.787329 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:21.788453 8827 raft_consensus.cc:1203] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:21.815662 6638 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:21.829517 11219 tablet_copy_client.cc:308] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:21.830572 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:51915)
I20240501 08:14:21.830763 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:51915: session id = 0a814bba1abe4653bcd418a83ec42370-9b9bd7500e8747c6b8ee94a1faa1c5f7
I20240501 08:14:21.831853 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:21.833740 6658 tablet_copy_source_session.cc:205] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:21.859122 11219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:21.932065 11219 tablet_copy_client.cc:783] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:21.932356 11219 tablet_copy_client.cc:647] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:21.935644 11219 tablet_copy_client.cc:523] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:21.942673 11219 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:21.943557 11078 ts_tablet_manager.cc:902] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:21.949952 6638 consensus_queue.cc:237] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
W20240501 08:14:21.954996 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:21.955809 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b9bd7500e8747c6b8ee94a1faa1c5f7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:21.957361 6899 raft_consensus.cc:1261] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:21.959105 11241 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:21.963882 7028 raft_consensus.cc:1261] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:21.964653 11241 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:21.974468 11153 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:21.979646 7028 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:21.986850 6494 catalog_manager.cc:5484] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:22.027244 6899 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:22.045876 8959 raft_consensus.cc:1203] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:22.143407 11078 tablet_copy_client.cc:308] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:22.144398 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet a9e70954aa4e4fcfbad12e8bfe3e0059 from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:54455)
I20240501 08:14:22.144593 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet a9e70954aa4e4fcfbad12e8bfe3e0059 from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:54455: session id = a518ca5e9de24204bd028b0ce4a77d58-a9e70954aa4e4fcfbad12e8bfe3e0059
I20240501 08:14:22.146883 7049 tablet_copy_source_session.cc:205] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:22.148556 11078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e70954aa4e4fcfbad12e8bfe3e0059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.185510 11078 tablet_copy_client.cc:783] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:22.185819 11078 tablet_copy_client.cc:647] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:22.190589 11078 tablet_copy_client.cc:523] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:22.195739 11078 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:22.244030 11278 ts_tablet_manager.cc:902] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:22.292236 11219 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:22.292758 11219 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:22.293053 11219 ts_tablet_manager.cc:1367] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.351s user 0.069s sys 0.049s
I20240501 08:14:22.294001 11219 raft_consensus.cc:348] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:22.304520 11219 raft_consensus.cc:729] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.305467 11219 consensus_queue.cc:260] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:22.317767 11219 ts_tablet_manager.cc:1398] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20240501 08:14:22.328501 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-9b9bd7500e8747c6b8ee94a1faa1c5f7 received from {username='slave'} at 127.25.107.5:51915
I20240501 08:14:22.328687 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-9b9bd7500e8747c6b8ee94a1faa1c5f7 on tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:22.340641 11078 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:22.341145 11078 tablet_bootstrap.cc:492] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:22.341441 11078 ts_tablet_manager.cc:1367] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.146s user 0.042s sys 0.012s
I20240501 08:14:22.341625 6769 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:22.342289 11078 raft_consensus.cc:348] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:22.342554 11078 raft_consensus.cc:729] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.342832 11078 consensus_queue.cc:260] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:22.347935 6899 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:14:22.348826 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a56c284444496b8579600a49f19d1d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:22.349196 10608 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:22.352275 7028 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:22.353024 10608 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:22.359475 8959 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:22.359946 11078 ts_tablet_manager.cc:1398] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.018s user 0.000s sys 0.004s
I20240501 08:14:22.360437 11309 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:22.361003 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-a9e70954aa4e4fcfbad12e8bfe3e0059 received from {username='slave'} at 127.25.107.7:54455
I20240501 08:14:22.361156 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-a9e70954aa4e4fcfbad12e8bfe3e0059 on tablet a9e70954aa4e4fcfbad12e8bfe3e0059 with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:22.362708 10608 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:22.368618 6899 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:22.376435 6494 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:22.409682 8959 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:22.417028 7028 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:22.426367 11278 tablet_copy_client.cc:308] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:22.427520 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 45d076303ea24d7fa0fed5a9d132019c from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:46523)
I20240501 08:14:22.427683 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 45d076303ea24d7fa0fed5a9d132019c from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:46523: session id = aba7c226aaca4e8c90931f547d48aaa3-45d076303ea24d7fa0fed5a9d132019c
I20240501 08:14:22.431625 7049 tablet_copy_source_session.cc:205] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:22.433430 11278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d076303ea24d7fa0fed5a9d132019c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.450075 11278 tablet_copy_client.cc:783] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:22.450573 11278 tablet_copy_client.cc:647] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:22.452417 11278 tablet_copy_client.cc:523] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:22.456149 11278 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:22.459975 8827 raft_consensus.cc:1203] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:14:22.586392 8561 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:22.596763 11369 ts_tablet_manager.cc:902] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:22.632287 6899 consensus_queue.cc:237] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:22.638643 11278 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:22.639305 11278 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:22.639632 11278 ts_tablet_manager.cc:1367] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.184s user 0.047s sys 0.009s
I20240501 08:14:22.643215 11278 raft_consensus.cc:348] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:22.643525 11278 raft_consensus.cc:729] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.644245 11369 tablet_copy_client.cc:308] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:22.644197 11278 consensus_queue.cc:260] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:22.645064 6638 raft_consensus.cc:1261] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:22.645304 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 658f2a172a2d47c8bf8b44c9d51132e4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:22.645695 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:33725)
I20240501 08:14:22.645820 10671 consensus_queue.cc:1035] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:22.645916 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:33725: session id = aba7c226aaca4e8c90931f547d48aaa3-9b9bd7500e8747c6b8ee94a1faa1c5f7
I20240501 08:14:22.646958 6638 consensus_queue.cc:237] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:22.649210 6658 tablet_copy_source_session.cc:205] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:22.651103 11369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.652580 6769 raft_consensus.cc:1261] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:22.653272 10671 consensus_queue.cc:1035] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:22.660852 11278 ts_tablet_manager.cc:1398] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20240501 08:14:22.662397 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-45d076303ea24d7fa0fed5a9d132019c received from {username='slave'} at 127.25.107.6:46523
I20240501 08:14:22.662603 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-45d076303ea24d7fa0fed5a9d132019c on tablet 45d076303ea24d7fa0fed5a9d132019c with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:22.662925 7028 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:22.664069 10577 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:14:22.664781 6572 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc19d80bad46460087c143778a4b0638. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:22.668076 6768 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:22.668882 11153 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:22.672627 11369 tablet_copy_client.cc:783] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:22.672997 11369 tablet_copy_client.cc:647] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:22.676043 8695 raft_consensus.cc:1203] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:22.678503 11369 tablet_copy_client.cc:523] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:22.682229 11369 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:22.728528 11153 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:22.731086 7029 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:22.747759 6494 catalog_manager.cc:5484] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:22.766988 6769 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:14:22.782924 6571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 43724.3us
I20240501 08:14:22.794967 11399 raft_consensus.cc:2932] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:22.812425 6637 raft_consensus.cc:2932] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:22.815389 6769 raft_consensus.cc:2932] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:22.831583 6488 catalog_manager.cc:5484] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:22.856590 6769 consensus_queue.cc:237] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:22.875375 7028 raft_consensus.cc:1261] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:22.875433 8827 raft_consensus.cc:1261] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:14:22.875595 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdaad8e1d86942ddbc97958dcf31b344. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:22.876066 11310 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:22.876596 11310 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:22.877164 6637 raft_consensus.cc:1261] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:22.878962 11310 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:22.903664 11369 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:22.904167 11369 tablet_bootstrap.cc:492] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:22.904467 11369 ts_tablet_manager.cc:1367] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.222s user 0.084s sys 0.044s
I20240501 08:14:22.905351 11369 raft_consensus.cc:348] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:22.905627 11369 raft_consensus.cc:729] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.905900 11369 consensus_queue.cc:260] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:22.906499 11369 ts_tablet_manager.cc:1398] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:14:22.907305 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-9b9bd7500e8747c6b8ee94a1faa1c5f7 received from {username='slave'} at 127.25.107.6:33725
I20240501 08:14:22.907442 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-9b9bd7500e8747c6b8ee94a1faa1c5f7 on tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:22.913969 11310 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:22.918417 6637 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:22.922462 7028 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:22.930560 6488 catalog_manager.cc:5484] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:22.953423 8827 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:23.024675 8695 raft_consensus.cc:1203] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:23.031570 11440 ts_tablet_manager.cc:902] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:23.050105 11369 ts_tablet_manager.cc:902] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:23.099042 11278 ts_tablet_manager.cc:902] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:23.106693 11369 tablet_copy_client.cc:308] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:14:23.107573 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 658f2a172a2d47c8bf8b44c9d51132e4 from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:40161)
I20240501 08:14:23.107774 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 658f2a172a2d47c8bf8b44c9d51132e4 from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:40161: session id = aba7c226aaca4e8c90931f547d48aaa3-658f2a172a2d47c8bf8b44c9d51132e4
I20240501 08:14:23.111017 6919 tablet_copy_source_session.cc:205] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:23.112869 11440 tablet_copy_client.cc:308] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:23.113950 11369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658f2a172a2d47c8bf8b44c9d51132e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.114449 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet a7a56c284444496b8579600a49f19d1d from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:49989)
I20240501 08:14:23.114600 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet a7a56c284444496b8579600a49f19d1d from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:49989: session id = a518ca5e9de24204bd028b0ce4a77d58-a7a56c284444496b8579600a49f19d1d
I20240501 08:14:23.117483 6789 tablet_copy_source_session.cc:205] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:23.119450 11440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a56c284444496b8579600a49f19d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.139256 11278 tablet_copy_client.cc:308] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:23.140264 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet dc19d80bad46460087c143778a4b0638 from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:33725)
I20240501 08:14:23.140455 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet dc19d80bad46460087c143778a4b0638 from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:33725: session id = aba7c226aaca4e8c90931f547d48aaa3-dc19d80bad46460087c143778a4b0638
I20240501 08:14:23.143291 6658 tablet_copy_source_session.cc:205] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:23.144826 11278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc19d80bad46460087c143778a4b0638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.151947 11278 tablet_copy_client.cc:783] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:23.152240 11278 tablet_copy_client.cc:647] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:23.160596 11278 tablet_copy_client.cc:523] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:23.179446 11278 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:23.186309 11369 tablet_copy_client.cc:783] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:23.190584 11369 tablet_copy_client.cc:647] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:23.195384 11369 tablet_copy_client.cc:523] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:23.200871 11369 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:23.203589 11440 tablet_copy_client.cc:783] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:23.204192 11440 tablet_copy_client.cc:647] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:23.224725 11440 tablet_copy_client.cc:523] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:23.250489 11278 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:23.251121 11278 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:23.251417 11278 ts_tablet_manager.cc:1367] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.072s user 0.044s sys 0.010s
I20240501 08:14:23.252440 11278 raft_consensus.cc:348] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:23.252709 11278 raft_consensus.cc:729] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.252990 11278 consensus_queue.cc:260] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:23.253707 11278 ts_tablet_manager.cc:1398] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:23.255028 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-dc19d80bad46460087c143778a4b0638 received from {username='slave'} at 127.25.107.6:33725
I20240501 08:14:23.255205 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-dc19d80bad46460087c143778a4b0638 on tablet dc19d80bad46460087c143778a4b0638 with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:23.258196 11440 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:23.300413 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:23.307492 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:23.366714 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:23.373966 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:23.406553 6637 consensus_queue.cc:237] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
W20240501 08:14:23.409154 6571 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc19d80bad46460087c143778a4b0638. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:23.410933 8695 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:23.410920 7028 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:23.412137 6769 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:23.412743 10577 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:23.413251 10577 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:23.413652 10577 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:23.418834 10577 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:23.422072 7028 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:23.462689 6769 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:23.464887 6496 catalog_manager.cc:5484] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:23.481870 11516 ts_tablet_manager.cc:902] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:23.486029 7028 consensus_queue.cc:237] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:23.488322 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:23.493050 8694 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:23.501674 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:23.503088 8694 raft_consensus.cc:1261] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:23.504171 11227 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:23.505270 6637 raft_consensus.cc:1261] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:23.505795 11218 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:14:23.506114 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60238162349647c681685349c6f8ede4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:23.515961 6899 raft_consensus.cc:1261] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:23.516647 11227 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:23.524344 11218 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:23.528048 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:23.534026 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:23.559696 6637 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:23.566849 8694 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:23.593451 6899 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:23.607811 6491 catalog_manager.cc:5484] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 4 to 5, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:23.638769 11516 tablet_copy_client.cc:308] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:23.639775 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet fdaad8e1d86942ddbc97958dcf31b344 from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:55445)
I20240501 08:14:23.640051 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet fdaad8e1d86942ddbc97958dcf31b344 from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:55445: session id = 0a814bba1abe4653bcd418a83ec42370-fdaad8e1d86942ddbc97958dcf31b344
I20240501 08:14:23.644157 6789 tablet_copy_source_session.cc:205] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:23.651739 11516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdaad8e1d86942ddbc97958dcf31b344. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.667629 11516 tablet_copy_client.cc:783] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:23.667955 11516 tablet_copy_client.cc:647] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:23.680635 11516 tablet_copy_client.cc:523] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:23.688289 11516 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:23.678221 11369 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:23.710233 11369 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:23.710582 11369 ts_tablet_manager.cc:1367] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.510s user 0.046s sys 0.013s
I20240501 08:14:23.712409 11369 raft_consensus.cc:348] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:23.718966 11369 raft_consensus.cc:729] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.721598 11369 consensus_queue.cc:260] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:23.725595 11369 ts_tablet_manager.cc:1398] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 08:14:23.734393 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-658f2a172a2d47c8bf8b44c9d51132e4 received from {username='slave'} at 127.25.107.6:40161
I20240501 08:14:23.735328 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-658f2a172a2d47c8bf8b44c9d51132e4 on tablet 658f2a172a2d47c8bf8b44c9d51132e4 with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:23.768868 11440 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:23.769719 11440 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:23.771449 11440 ts_tablet_manager.cc:1367] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.513s user 0.071s sys 0.000s
I20240501 08:14:23.773192 11440 raft_consensus.cc:348] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:23.773525 11440 raft_consensus.cc:729] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.775171 11440 consensus_queue.cc:260] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:23.782347 11440 ts_tablet_manager.cc:1398] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20240501 08:14:23.783171 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-a7a56c284444496b8579600a49f19d1d received from {username='slave'} at 127.25.107.7:49989
I20240501 08:14:23.783313 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-a7a56c284444496b8579600a49f19d1d on tablet a7a56c284444496b8579600a49f19d1d with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:23.838407 8694 raft_consensus.cc:1203] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:23.875926 11516 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:23.876819 11516 tablet_bootstrap.cc:492] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:23.877163 11516 ts_tablet_manager.cc:1367] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.189s user 0.055s sys 0.001s
I20240501 08:14:23.878172 11516 raft_consensus.cc:348] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:23.878561 11516 raft_consensus.cc:729] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.878871 11516 consensus_queue.cc:260] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:23.886380 11516 ts_tablet_manager.cc:1398] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240501 08:14:23.890077 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-fdaad8e1d86942ddbc97958dcf31b344 received from {username='slave'} at 127.25.107.5:55445
I20240501 08:14:23.890285 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-fdaad8e1d86942ddbc97958dcf31b344 on tablet fdaad8e1d86942ddbc97958dcf31b344 with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:23.913209 6637 consensus_queue.cc:237] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:23.953895 6769 raft_consensus.cc:1261] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:23.954123 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37a6157d7d734a80900139944481928c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:23.955088 10577 consensus_queue.cc:1035] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:23.959089 6899 raft_consensus.cc:1261] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:23.961166 11519 consensus_queue.cc:1035] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:24.027426 11610 ts_tablet_manager.cc:902] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
W20240501 08:14:24.029817 6705 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 83443.6us
I20240501 08:14:24.031548 11612 raft_consensus.cc:2932] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:24.032879 8561 raft_consensus.cc:1203] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:24.035640 11440 ts_tablet_manager.cc:902] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:24.041507 6899 raft_consensus.cc:2932] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:24.042982 6494 catalog_manager.cc:5484] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:24.041121 6769 raft_consensus.cc:2932] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:24.074031 8826 raft_consensus.cc:1203] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:24.152235 11610 tablet_copy_client.cc:308] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:24.153170 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 60238162349647c681685349c6f8ede4 from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:37833)
I20240501 08:14:24.153363 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 60238162349647c681685349c6f8ede4 from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:37833: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-60238162349647c681685349c6f8ede4
I20240501 08:14:24.156322 7049 tablet_copy_source_session.cc:205] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:24.165673 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:14:24.191398 6832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.1:45193, user_credentials={real_user=slave}} blocked reactor thread for 52379us
I20240501 08:14:24.214100 11610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60238162349647c681685349c6f8ede4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.225912 11440 tablet_copy_client.cc:308] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:24.227231 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet dc19d80bad46460087c143778a4b0638 from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:48573)
I20240501 08:14:24.227448 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet dc19d80bad46460087c143778a4b0638 from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:48573: session id = a518ca5e9de24204bd028b0ce4a77d58-dc19d80bad46460087c143778a4b0638
I20240501 08:14:24.231830 6658 tablet_copy_source_session.cc:205] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:24.239126 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.239485 11440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc19d80bad46460087c143778a4b0638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.255841 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.282032 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:24.294590 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:24.294800 11610 tablet_copy_client.cc:783] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:24.295199 11610 tablet_copy_client.cc:647] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:24.309497 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:24.328454 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.330638 11610 tablet_copy_client.cc:523] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:24.362856 11610 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:24.372663 11440 tablet_copy_client.cc:783] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:24.373009 11440 tablet_copy_client.cc:647] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:24.388528 11440 tablet_copy_client.cc:523] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:24.394630 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.396196 11673 ts_tablet_manager.cc:902] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:24.412148 11440 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:24.436417 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.450172 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.452379 11673 tablet_copy_client.cc:308] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:24.453277 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 37a6157d7d734a80900139944481928c from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:33725)
I20240501 08:14:24.453450 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 37a6157d7d734a80900139944481928c from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:33725: session id = aba7c226aaca4e8c90931f547d48aaa3-37a6157d7d734a80900139944481928c
I20240501 08:14:24.456756 6658 tablet_copy_source_session.cc:205] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:24.459206 11673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a6157d7d734a80900139944481928c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.463529 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:24.474467 11673 tablet_copy_client.cc:783] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:24.474908 11673 tablet_copy_client.cc:647] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:24.479491 11673 tablet_copy_client.cc:523] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:24.488476 11673 tablet_bootstrap.cc:492] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
I20240501 08:14:24.534358 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:24.547421 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.562602 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.584466 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:24.623675 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:24.767658 11440 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:24.768244 11440 tablet_bootstrap.cc:492] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:24.768573 11440 ts_tablet_manager.cc:1367] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.357s user 0.046s sys 0.006s
I20240501 08:14:24.769459 11440 raft_consensus.cc:348] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:24.769827 11440 raft_consensus.cc:729] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.770180 11440 consensus_queue.cc:260] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:24.778870 11440 ts_tablet_manager.cc:1398] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240501 08:14:24.779744 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-dc19d80bad46460087c143778a4b0638 received from {username='slave'} at 127.25.107.7:48573
I20240501 08:14:24.779887 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-dc19d80bad46460087c143778a4b0638 on tablet dc19d80bad46460087c143778a4b0638 with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:24.837787 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.864228 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.878224 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:24.901504 11673 tablet_bootstrap.cc:492] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:24.902093 11673 tablet_bootstrap.cc:492] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:24.902689 11673 ts_tablet_manager.cc:1367] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.414s user 0.076s sys 0.030s
I20240501 08:14:24.903834 11673 raft_consensus.cc:348] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:24.906088 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.912537 11610 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:24.912811 11673 raft_consensus.cc:729] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.913220 11673 consensus_queue.cc:260] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:24.917618 11673 ts_tablet_manager.cc:1398] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20240501 08:14:24.918707 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-37a6157d7d734a80900139944481928c received from {username='slave'} at 127.25.107.6:33725
I20240501 08:14:24.918879 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-37a6157d7d734a80900139944481928c on tablet 37a6157d7d734a80900139944481928c with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:24.919114 11610 tablet_bootstrap.cc:492] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:24.920882 11610 ts_tablet_manager.cc:1367] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.558s user 0.050s sys 0.002s
I20240501 08:14:24.921861 11610 raft_consensus.cc:348] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:24.922274 11610 raft_consensus.cc:729] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.922641 11610 consensus_queue.cc:260] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:24.935564 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:24.954061 11610 ts_tablet_manager.cc:1398] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.033s user 0.004s sys 0.004s
I20240501 08:14:24.955193 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-60238162349647c681685349c6f8ede4 received from {username='slave'} at 127.25.107.8:37833
I20240501 08:14:24.955338 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-60238162349647c681685349c6f8ede4 on tablet 60238162349647c681685349c6f8ede4 with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:24.969573 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:24.976584 8695 raft_consensus.cc:1203] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:25.004092 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:25.027235 8959 raft_consensus.cc:1203] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:25.073431 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:25.122720 8826 raft_consensus.cc:1203] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:25.255820 7028 consensus_queue.cc:237] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
W20240501 08:14:25.266341 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45d076303ea24d7fa0fed5a9d132019c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:25.267414 8695 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:25.268285 6637 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:25.268868 6768 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:25.268932 11546 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:25.269423 11544 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:25.269991 11544 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:25.273626 11544 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:25.277698 6769 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:25.283921 6494 catalog_manager.cc:5484] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:25.287005 6637 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:25.298506 8694 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:25.351433 6769 consensus_queue.cc:237] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:25.375942 8826 raft_consensus.cc:1261] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:25.376343 8561 raft_consensus.cc:1261] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:25.376550 6638 raft_consensus.cc:1261] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:25.377233 11615 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:25.377643 11310 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:25.378582 11310 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:25.386871 11615 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:25.391124 6638 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:25.400374 6493 catalog_manager.cc:5484] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:25.405565 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet fdaad8e1d86942ddbc97958dcf31b344 with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:25.409421 8561 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:25.438199 8826 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:25.450084 11820 tablet_replica.cc:330] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:25.450924 11820 raft_consensus.cc:2229] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:25.451665 11820 raft_consensus.cc:2258] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:25.492900 11820 ts_tablet_manager.cc:1875] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:25.537732 11820 ts_tablet_manager.cc:1888] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:25.538012 11820 log.cc:1196] T fdaad8e1d86942ddbc97958dcf31b344 P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/fdaad8e1d86942ddbc97958dcf31b344
I20240501 08:14:25.542047 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet fdaad8e1d86942ddbc97958dcf31b344 (table rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]) successfully deleted
I20240501 08:14:25.722714 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:25.790851 11888 ts_tablet_manager.cc:902] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:25.841563 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:25.842084 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:25.881935 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:25.904942 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:25.939893 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:25.981417 11888 tablet_copy_client.cc:308] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:25.982419 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 45d076303ea24d7fa0fed5a9d132019c from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:54455)
I20240501 08:14:25.982581 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 45d076303ea24d7fa0fed5a9d132019c from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:54455: session id = a518ca5e9de24204bd028b0ce4a77d58-45d076303ea24d7fa0fed5a9d132019c
I20240501 08:14:25.985451 7049 tablet_copy_source_session.cc:205] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:25.987216 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d076303ea24d7fa0fed5a9d132019c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:26.010329 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.011718 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.146103 11888 tablet_copy_client.cc:783] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:26.146540 11888 tablet_copy_client.cc:647] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:26.156020 11888 tablet_copy_client.cc:523] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:26.160032 11888 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:26.287354 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.288060 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:26.339038 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:26.344756 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:26.355486 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.380841 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.391731 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.399369 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.403620 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:26.444633 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:26.445379 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:26.462216 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.471451 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:14:26.475158 6962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 77534.8us
I20240501 08:14:26.481099 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.487834 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:26.488550 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:26.627221 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.654132 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.660265 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:26.692998 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:26.693405 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.725220 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:26.726703 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:14:26.750150 11888 log.cc:924] Time spent T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: Append to log took a long time: real 0.113s user 0.002s sys 0.000s
I20240501 08:14:26.773756 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:26.851001 6638 consensus_queue.cc:237] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:26.874945 8561 raft_consensus.cc:1261] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:26.876418 11867 consensus_queue.cc:1035] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:26.879137 6899 raft_consensus.cc:1261] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:26.884035 11612 consensus_queue.cc:1035] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:26.915189 11888 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:26.915666 11867 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:26.918820 11888 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:26.919159 11888 ts_tablet_manager.cc:1367] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.759s user 0.052s sys 0.003s
I20240501 08:14:26.919131 8563 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:26.919981 11888 raft_consensus.cc:348] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:26.920235 11888 raft_consensus.cc:729] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:26.920522 11888 consensus_queue.cc:260] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:26.925261 6496 catalog_manager.cc:5484] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:26.928485 6899 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:26.960980 11888 ts_tablet_manager.cc:1398] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.042s user 0.007s sys 0.001s
I20240501 08:14:26.961913 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-45d076303ea24d7fa0fed5a9d132019c received from {username='slave'} at 127.25.107.7:54455
I20240501 08:14:26.962074 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-45d076303ea24d7fa0fed5a9d132019c on tablet 45d076303ea24d7fa0fed5a9d132019c with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:26.990756 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db7f887af8f425c8fa661196add4d37 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:27.002285 12039 tablet_replica.cc:330] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:27.003026 12039 raft_consensus.cc:2229] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:27.003520 12039 raft_consensus.cc:2258] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:27.007476 12039 ts_tablet_manager.cc:1875] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:27.017719 8827 raft_consensus.cc:1203] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:27.023921 12039 ts_tablet_manager.cc:1888] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:27.024231 12039 log.cc:1196] T 4db7f887af8f425c8fa661196add4d37 P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/4db7f887af8f425c8fa661196add4d37
I20240501 08:14:27.025537 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 4db7f887af8f425c8fa661196add4d37 (table rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]) successfully deleted
I20240501 08:14:27.108407 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:27.131705 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:27.180065 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:27.204295 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:27.222494 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:27.327797 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:27.437215 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:27.456413 6638 consensus_queue.cc:237] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:27.462800 8827 raft_consensus.cc:1261] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:27.463217 6899 raft_consensus.cc:1261] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:27.463515 11612 consensus_queue.cc:1035] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:27.463878 11867 consensus_queue.cc:1035] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:27.505769 6638 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:27.515578 8563 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:27.516281 7029 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:27.516474 11612 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:27.517064 11612 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:27.531366 11867 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:27.532069 7029 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:27.539034 6496 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:27.552853 8563 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:27.553428 11612 raft_consensus.cc:2932] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:27.559473 8827 raft_consensus.cc:2932] T 78d1df7b0ea44f89b9f22c156f907c1e P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:27.576874 6898 raft_consensus.cc:2932] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:27.582875 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:27.584015 6488 catalog_manager.cc:5484] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:27.603986 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d1df7b0ea44f89b9f22c156f907c1e with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:27.604506 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:27.638664 12153 tablet_replica.cc:330] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:27.639073 12153 raft_consensus.cc:2229] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:27.639622 12154 tablet_replica.cc:330] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:27.639992 12154 raft_consensus.cc:2229] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:27.640391 12154 raft_consensus.cc:2258] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:27.650457 12153 raft_consensus.cc:2258] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:27.652243 12154 ts_tablet_manager.cc:1875] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:27.659021 12153 ts_tablet_manager.cc:1875] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:27.694362 12154 ts_tablet_manager.cc:1888] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:27.694672 12154 log.cc:1196] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:14:27.695761 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 30b4f075057e42d0bb3f8cdabdfbc643 (table rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]) successfully deleted
I20240501 08:14:27.712105 12153 ts_tablet_manager.cc:1888] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:27.712800 12153 log.cc:1196] T 78d1df7b0ea44f89b9f22c156f907c1e P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/78d1df7b0ea44f89b9f22c156f907c1e
I20240501 08:14:27.721191 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet 78d1df7b0ea44f89b9f22c156f907c1e (table rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]) successfully deleted
I20240501 08:14:27.783736 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:27.797168 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:27.799192 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:27.816839 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:27.893638 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:27.921977 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:27.932211 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:27.937124 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:28.142658 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:28.155561 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:28.188982 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:28.200407 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:28.232687 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:28.244110 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:28.290544 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:28.300333 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:28.305876 6637 consensus_queue.cc:237] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:28.323680 8563 raft_consensus.cc:1261] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:28.324368 11867 consensus_queue.cc:1035] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:28.355134 11867 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:28.358619 8561 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:28.371568 6494 catalog_manager.cc:5484] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:28.411748 6637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643"
dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33496
I20240501 08:14:28.412128 6637 raft_consensus.cc:595] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Received request to transfer leadership
W20240501 08:14:28.476480 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 63552.4us
I20240501 08:14:28.524437 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db7f887af8f425c8fa661196add4d37 with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:28.525482 12287 tablet_replica.cc:330] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:28.525985 12287 raft_consensus.cc:2229] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:28.526403 12287 raft_consensus.cc:2258] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:28.527151 6637 consensus_queue.cc:237] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:28.528218 12287 ts_tablet_manager.cc:1875] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:28.531025 8561 raft_consensus.cc:1261] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:14:28.532413 6573 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): Couldn't send request to peer 28b7613bd4834f3ab245c82dd7e80240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4db7f887af8f425c8fa661196add4d37. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:28.533202 11867 consensus_queue.cc:1035] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:28.535648 12287 ts_tablet_manager.cc:1888] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:28.536394 12287 log.cc:1196] T 4db7f887af8f425c8fa661196add4d37 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/4db7f887af8f425c8fa661196add4d37
I20240501 08:14:28.541661 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 4db7f887af8f425c8fa661196add4d37 (table rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]) successfully deleted
I20240501 08:14:28.546913 11612 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:14:28.555444 8563 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:14:28.578496 11867 raft_consensus.cc:979] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: : Instructing follower 28b7613bd4834f3ab245c82dd7e80240 to start an election
I20240501 08:14:28.578716 11867 raft_consensus.cc:1067] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Signalling peer 28b7613bd4834f3ab245c82dd7e80240 to start an election
I20240501 08:14:28.580210 7028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643"
dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
from {username='slave'} at 127.25.107.1:45633
I20240501 08:14:28.580543 7028 raft_consensus.cc:482] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:28.580687 7028 raft_consensus.cc:3037] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:28.596379 6494 catalog_manager.cc:5484] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 6 to 7, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:14:28.584383 7028 raft_consensus.cc:504] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:28.603843 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:28.604166 6637 raft_consensus.cc:3032] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:28.604288 6637 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:28.604633 6637 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:28.606985 6637 raft_consensus.cc:3037] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:28.630157 6637 raft_consensus.cc:2454] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 2.
I20240501 08:14:28.631136 6962 leader_election.cc:302] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:14:28.631634 12266 raft_consensus.cc:2781] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:28.631790 12266 raft_consensus.cc:686] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:14:28.636399 7028 leader_election.cc:288] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 2 election: Requested vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:14:28.637665 8563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0a814bba1abe4653bcd418a83ec42370"
I20240501 08:14:28.638119 8563 raft_consensus.cc:3037] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:28.644420 8563 raft_consensus.cc:2454] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 2.
I20240501 08:14:28.645236 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4db7f887af8f425c8fa661196add4d37 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:28.632136 12266 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:28.740221 6494 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 1 to 2, leader changed from 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 2 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:28.795451 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:28.839010 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:28.848659 6616 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:28.860863 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:28.905604 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:28.956348 12348 ts_tablet_manager.cc:902] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:28.983503 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:28.991190 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:29.012172 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:29.017465 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:29.073170 12348 tablet_copy_client.cc:308] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:14:29.074357 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 4db7f887af8f425c8fa661196add4d37 from peer 28b7613bd4834f3ab245c82dd7e80240 ({username='slave'} at 127.25.107.4:37539)
I20240501 08:14:29.074570 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 4db7f887af8f425c8fa661196add4d37 from peer 28b7613bd4834f3ab245c82dd7e80240 at {username='slave'} at 127.25.107.4:37539: session id = 28b7613bd4834f3ab245c82dd7e80240-4db7f887af8f425c8fa661196add4d37
I20240501 08:14:29.077713 6658 tablet_copy_source_session.cc:205] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:29.084491 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:29.085796 12348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db7f887af8f425c8fa661196add4d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:29.093109 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:29.112360 12348 tablet_copy_client.cc:783] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:29.114300 12348 tablet_copy_client.cc:647] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:29.118424 12348 tablet_copy_client.cc:523] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:29.141417 6616 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:29.143831 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:29.144457 12266 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:29.144778 12348 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:14:29.172304 6768 consensus_queue.cc:237] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:29.202461 8563 raft_consensus.cc:1261] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:29.203250 11808 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:29.211509 8826 raft_consensus.cc:1261] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:29.213559 11808 consensus_queue.cc:1035] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:29.222957 8561 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:29.223642 12266 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:29.260488 11808 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:29.261969 8827 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:29.265211 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:29.277308 6494 catalog_manager.cc:5484] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 6 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:29.283660 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:29.306607 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:29.309944 8563 raft_consensus.cc:2932] T fdaad8e1d86942ddbc97958dcf31b344 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:29.357586 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:29.389519 6616 tablet_service.cc:1514] Processing DeleteTablet for tablet fdaad8e1d86942ddbc97958dcf31b344 with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37350
I20240501 08:14:29.425014 12405 tablet_replica.cc:330] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:14:29.425798 12405 raft_consensus.cc:2229] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:29.426337 12405 raft_consensus.cc:2258] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:29.429832 12405 ts_tablet_manager.cc:1875] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:29.446151 12348 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:29.450160 12348 tablet_bootstrap.cc:492] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap complete.
I20240501 08:14:29.450634 12348 ts_tablet_manager.cc:1367] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.308s user 0.051s sys 0.010s
I20240501 08:14:29.451799 12348 raft_consensus.cc:348] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:29.452154 12348 raft_consensus.cc:729] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:14:29.452450 12348 consensus_queue.cc:260] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:14:29.453469 12348 ts_tablet_manager.cc:1398] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:14:29.454728 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session 28b7613bd4834f3ab245c82dd7e80240-4db7f887af8f425c8fa661196add4d37 received from {username='slave'} at 127.25.107.4:37539
I20240501 08:14:29.467311 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session 28b7613bd4834f3ab245c82dd7e80240-4db7f887af8f425c8fa661196add4d37 on tablet 4db7f887af8f425c8fa661196add4d37 with peer 28b7613bd4834f3ab245c82dd7e80240
I20240501 08:14:29.494603 12405 ts_tablet_manager.cc:1888] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:14:29.498489 12405 log.cc:1196] T fdaad8e1d86942ddbc97958dcf31b344 P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/fdaad8e1d86942ddbc97958dcf31b344
I20240501 08:14:29.500052 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet fdaad8e1d86942ddbc97958dcf31b344 (table rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]) successfully deleted
I20240501 08:14:29.523087 7029 raft_consensus.cc:1203] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:14:29.707191 6636 consensus_queue.cc:237] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:29.754560 8825 raft_consensus.cc:1261] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:29.755471 12295 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:29.766541 8694 raft_consensus.cc:1261] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:29.767730 12295 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:29.769809 6769 raft_consensus.cc:1261] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:29.770560 12295 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:29.798430 12295 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:29.803867 6769 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:29.803993 8694 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:29.809813 8825 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:29.814517 6488 catalog_manager.cc:5484] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:29.819343 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet 29506d751571401f86bced7e6f5415d7 with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:29.834028 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:29.846022 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:29.846560 6616 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:29.858791 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:29.877771 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:29.893687 12491 tablet_replica.cc:330] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:29.893963 12491 raft_consensus.cc:2229] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:29.894402 12491 raft_consensus.cc:2258] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:29.895246 12491 ts_tablet_manager.cc:1875] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:29.929467 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:29.936949 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:29.941735 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:29.948815 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:29.960614 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:30.000419 6616 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:30.014855 12491 ts_tablet_manager.cc:1888] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:30.015111 12491 log.cc:1196] T 29506d751571401f86bced7e6f5415d7 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/29506d751571401f86bced7e6f5415d7
I20240501 08:14:30.015926 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 29506d751571401f86bced7e6f5415d7 (table rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]) successfully deleted
I20240501 08:14:30.038591 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:30.101671 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:30.111799 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:30.127846 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:30.138649 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:30.173835 7029 consensus_queue.cc:237] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:30.192734 6636 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:30.192888 6769 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:30.193336 8827 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:30.193452 12268 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:30.194788 11979 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:30.197690 12266 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:30.280316 12266 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:30.283468 8827 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:30.291183 6769 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:30.293056 6488 catalog_manager.cc:5484] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:30.301898 6636 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:30.353279 8675 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d076303ea24d7fa0fed5a9d132019c with delete_type TABLET_DATA_TOMBSTONED (TS aba7c226aaca4e8c90931f547d48aaa3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57258
I20240501 08:14:30.364820 12538 tablet_replica.cc:330] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: stopping tablet replica
I20240501 08:14:30.365605 12538 raft_consensus.cc:2229] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:30.366274 12538 raft_consensus.cc:2258] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:30.367794 12538 ts_tablet_manager.cc:1875] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:30.407511 12538 ts_tablet_manager.cc:1888] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:30.408485 12538 log.cc:1196] T 45d076303ea24d7fa0fed5a9d132019c P aba7c226aaca4e8c90931f547d48aaa3: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-5/wal/wals/45d076303ea24d7fa0fed5a9d132019c
I20240501 08:14:30.410096 6482 catalog_manager.cc:4830] TS aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): tablet 45d076303ea24d7fa0fed5a9d132019c (table rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]) successfully deleted
I20240501 08:14:30.450443 6768 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c"
dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39688
I20240501 08:14:30.450738 6768 raft_consensus.cc:595] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Received request to transfer leadership
I20240501 08:14:30.475451 12510 raft_consensus.cc:979] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: : Instructing follower 97cdee0bf9d84645a1451f9822c5d699 to start an election
I20240501 08:14:30.476203 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:30.476218 11819 raft_consensus.cc:1067] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Signalling peer 97cdee0bf9d84645a1451f9822c5d699 to start an election
I20240501 08:14:30.481155 6636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c"
dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
from {username='slave'} at 127.25.107.2:37487
I20240501 08:14:30.481480 6636 raft_consensus.cc:482] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:30.481614 6636 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:30.497753 6636 raft_consensus.cc:504] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:30.499228 6636 leader_election.cc:288] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 2 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219), a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929)
I20240501 08:14:30.499537 7029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:30.499792 7029 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:30.499969 8825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a518ca5e9de24204bd028b0ce4a77d58"
I20240501 08:14:30.500331 8825 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:30.503404 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:30.503660 6768 raft_consensus.cc:3032] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:30.503770 6768 raft_consensus.cc:729] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:14:30.503757 8825 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 2.
I20240501 08:14:30.504106 6768 consensus_queue.cc:260] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:30.504709 6768 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:30.506824 8563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "0a814bba1abe4653bcd418a83ec42370"
I20240501 08:14:30.507068 8563 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:30.510794 7029 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 2.
I20240501 08:14:30.513408 6768 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 2.
I20240501 08:14:30.514137 6574 leader_election.cc:302] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699, a518ca5e9de24204bd028b0ce4a77d58, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:14:30.514583 12290 raft_consensus.cc:2781] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:30.514719 12290 raft_consensus.cc:686] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 2 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:30.515081 12290 consensus_queue.cc:237] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:30.517215 8563 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 2.
W20240501 08:14:30.551276 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 59385.9us
I20240501 08:14:30.552659 8695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "aba7c226aaca4e8c90931f547d48aaa3"
I20240501 08:14:30.553050 8695 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:30.556497 8695 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 2.
I20240501 08:14:30.566522 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:30.590598 6494 catalog_manager.cc:5484] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 1 to 2, leader changed from cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 2 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:30.608598 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:30.659703 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:30.671133 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:30.681313 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:30.691334 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:30.823855 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:30.893625 8695 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:30.894423 12577 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:30.919528 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:30.926518 8563 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:30.928601 12577 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:30.939797 8825 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:30.940881 12290 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:30.944798 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:30.977685 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:30.983145 6616 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:31.065996 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:31.083488 7029 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:31.084410 12290 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:31.099920 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:31.103924 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:31.116531 6768 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:31.117795 12290 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:31.137660 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:31.306519 7029 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:31.339241 8563 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:31.340036 11979 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:31.364637 12268 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:31.377238 6490 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 5 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 2 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:31.404667 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:31.426410 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:31.478145 6616 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37350
I20240501 08:14:31.479916 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:31.479869 7029 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:31.480394 8563 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
W20240501 08:14:31.482426 6962 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): Couldn't send request to peer e3a1c4a5b44340be8a183d5d649b27ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30b4f075057e42d0bb3f8cdabdfbc643. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:31.505570 8561 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:31.510434 12701 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:31.542805 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:31.544737 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:31.566629 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:31.585062 12719 tablet_replica.cc:330] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:14:31.586304 12719 raft_consensus.cc:2229] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:31.589058 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:31.589566 12719 raft_consensus.cc:2258] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:31.592362 12719 ts_tablet_manager.cc:1875] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:31.595109 6618 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:31.645112 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:31.666280 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:31.679556 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:31.708106 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:31.717124 12701 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:31.728123 12719 ts_tablet_manager.cc:1888] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:14:31.728286 12719 log.cc:1196] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:14:31.728622 6490 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New cstate: current_term: 2 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:31.731688 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet 30b4f075057e42d0bb3f8cdabdfbc643 (table rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]) successfully deleted
I20240501 08:14:31.733624 6636 consensus_queue.cc:237] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:31.735564 6483 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:31.747648 8958 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:31.748598 12577 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:31.753108 8563 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:31.754725 12577 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:14:31.761893 6965 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 45047.9us
I20240501 08:14:31.774152 8563 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:31.786226 12577 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:14:31.791304 6573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 46009.7us
I20240501 08:14:31.795485 8958 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:31.798547 6618 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:31.805982 6490 catalog_manager.cc:5484] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:31.814991 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet 491674dfbf74498e8a787bebaa5e95da with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:31.819023 8561 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:31.821815 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 08:14:31.831974 6702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 66555.5us
I20240501 08:14:31.837685 12765 tablet_replica.cc:330] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:31.837940 12765 raft_consensus.cc:2229] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:31.838310 12765 raft_consensus.cc:2258] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:31.843780 12765 ts_tablet_manager.cc:1875] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:31.851840 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:31.870043 12765 ts_tablet_manager.cc:1888] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:31.871928 6768 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:31.874903 12765 log.cc:1196] T 491674dfbf74498e8a787bebaa5e95da P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/491674dfbf74498e8a787bebaa5e95da
I20240501 08:14:31.876156 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet 491674dfbf74498e8a787bebaa5e95da (table rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]) successfully deleted
I20240501 08:14:31.878584 12295 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:14:31.904867 6705 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 161249us
I20240501 08:14:31.921515 6768 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:14:31.927608 6571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 81416.4us
I20240501 08:14:31.963532 7028 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:31.998242 12786 ts_tablet_manager.cc:902] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:32.001888 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:32.035133 8958 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:32.035804 12701 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:32.037940 8561 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:32.038663 12268 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:32.042176 12268 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:32.046290 8561 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:32.053107 6494 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:32.060464 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f090fda814f455a88dccd6680bdd05f with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37350
I20240501 08:14:32.060889 12719 tablet_replica.cc:330] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:14:32.061103 12719 raft_consensus.cc:2229] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:32.061324 12719 raft_consensus.cc:2258] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:32.062089 12719 ts_tablet_manager.cc:1875] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:32.064636 6899 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:32.068462 12701 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:32.075222 8958 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:32.088243 12719 ts_tablet_manager.cc:1888] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:32.092448 12719 log.cc:1196] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f090fda814f455a88dccd6680bdd05f
I20240501 08:14:32.093472 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet 5f090fda814f455a88dccd6680bdd05f (table rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]) successfully deleted
I20240501 08:14:32.106838 6899 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:32.175065 12786 tablet_copy_client.cc:308] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:32.177413 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer e3a1c4a5b44340be8a183d5d649b27ac ({username='slave'} at 127.25.107.3:56639)
I20240501 08:14:32.177635 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer e3a1c4a5b44340be8a183d5d649b27ac at {username='slave'} at 127.25.107.3:56639: session id = e3a1c4a5b44340be8a183d5d649b27ac-30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:14:32.180387 7049 tablet_copy_source_session.cc:205] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:32.182616 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4f075057e42d0bb3f8cdabdfbc643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:32.201315 12786 tablet_copy_client.cc:783] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:32.201664 12786 tablet_copy_client.cc:647] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:32.203991 12786 tablet_copy_client.cc:523] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:32.208266 12786 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:32.322762 12786 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:32.323367 12786 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap complete.
I20240501 08:14:32.323714 12786 ts_tablet_manager.cc:1367] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.116s user 0.051s sys 0.008s
I20240501 08:14:32.324625 12786 raft_consensus.cc:348] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:32.324898 12786 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:32.325199 12786 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:32.329136 12786 ts_tablet_manager.cc:1398] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240501 08:14:32.333032 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-30b4f075057e42d0bb3f8cdabdfbc643 received from {username='slave'} at 127.25.107.3:56639
I20240501 08:14:32.333195 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-30b4f075057e42d0bb3f8cdabdfbc643 on tablet 30b4f075057e42d0bb3f8cdabdfbc643 with peer e3a1c4a5b44340be8a183d5d649b27ac
I20240501 08:14:32.477185 6768 consensus_queue.cc:237] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:32.492866 8561 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:32.494851 12753 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:32.496860 6898 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:32.497704 12753 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:32.528523 12896 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:32.529454 6898 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:32.535305 6899 raft_consensus.cc:1203] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 08:14:32.549207 6490 catalog_manager.cc:5484] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:32.554934 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet d7903106d7e74e1ea102b1abdbfce63e with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:32.555842 12906 tablet_replica.cc:330] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:32.556362 12906 raft_consensus.cc:2229] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:32.556682 12906 raft_consensus.cc:2258] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:32.557528 12906 ts_tablet_manager.cc:1875] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:32.562336 8561 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:32.575716 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:32.582461 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:32.612856 6636 consensus_queue.cc:237] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:32.624095 8825 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:32.624483 12906 ts_tablet_manager.cc:1888] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:32.624725 12906 log.cc:1196] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/d7903106d7e74e1ea102b1abdbfce63e
I20240501 08:14:32.624924 12577 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:32.634172 8694 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:32.634629 8561 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:32.639051 12757 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:32.639758 12757 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:32.639927 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet d7903106d7e74e1ea102b1abdbfce63e (table rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]) successfully deleted
I20240501 08:14:32.642670 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:32.669639 7029 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:32.670468 12577 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:32.718626 12577 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:32.722134 8563 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:32.730765 8825 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:32.741237 8694 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:32.743599 6496 catalog_manager.cc:5484] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 reported cstate change: config changed from index 6 to 8, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 2 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:32.750730 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet a25e1ddb987b4f8c8a4031053f581c7c with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:32.756723 7029 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:32.785856 12940 tablet_replica.cc:330] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:32.797859 6618 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:32.786190 12940 raft_consensus.cc:2229] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:32.810895 12940 raft_consensus.cc:2258] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:32.811828 12940 ts_tablet_manager.cc:1875] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:32.821636 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:32.844308 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:32.852373 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:32.863461 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:32.925848 12940 ts_tablet_manager.cc:1888] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 08:14:32.931982 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:32.963246 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:32.990339 12940 log.cc:1196] T a25e1ddb987b4f8c8a4031053f581c7c P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/a25e1ddb987b4f8c8a4031053f581c7c
I20240501 08:14:32.991809 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet a25e1ddb987b4f8c8a4031053f581c7c (table rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]) successfully deleted
I20240501 08:14:33.008383 6618 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:33.043016 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:33.045894 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:33.107399 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:33.168740 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:33.184445 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:33.302877 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:33.328006 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:33.357882 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:33.403316 6618 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:14:33.439908 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:33.443607 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:33.492996 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:33.503808 6637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "070e927b5d684367ae0b40b12a23da54"
dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33706
I20240501 08:14:33.504087 6637 raft_consensus.cc:595] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:14:33.528196 8805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:33.619004 12577 raft_consensus.cc:979] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: : Instructing follower a518ca5e9de24204bd028b0ce4a77d58 to start an election
I20240501 08:14:33.619179 12577 raft_consensus.cc:1067] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Signalling peer a518ca5e9de24204bd028b0ce4a77d58 to start an election
I20240501 08:14:33.619982 8825 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "070e927b5d684367ae0b40b12a23da54"
dest_uuid: "a518ca5e9de24204bd028b0ce4a77d58"
from {username='slave'} at 127.25.107.1:59341
I20240501 08:14:33.620314 8825 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:33.620472 8825 raft_consensus.cc:3037] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:33.623756 8825 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:33.638063 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070e927b5d684367ae0b40b12a23da54" candidate_uuid: "a518ca5e9de24204bd028b0ce4a77d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:33.638372 6768 raft_consensus.cc:3037] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:33.641275 6768 raft_consensus.cc:2454] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a518ca5e9de24204bd028b0ce4a77d58 in term 2.
I20240501 08:14:33.645690 7029 consensus_queue.cc:237] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:33.655463 8825 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [CANDIDATE]: Term 2 election: Requested vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:14:33.655565 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070e927b5d684367ae0b40b12a23da54" candidate_uuid: "a518ca5e9de24204bd028b0ce4a77d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:33.655776 6637 raft_consensus.cc:3032] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:33.655839 6637 raft_consensus.cc:729] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:33.656108 6637 consensus_queue.cc:260] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:33.656577 6637 raft_consensus.cc:3037] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:33.659597 6637 raft_consensus.cc:2454] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a518ca5e9de24204bd028b0ce4a77d58 in term 2.
I20240501 08:14:33.660571 8760 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, a518ca5e9de24204bd028b0ce4a77d58, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:14:33.701589 8958 raft_consensus.cc:1261] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:33.702440 12268 consensus_queue.cc:1035] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:33.710487 6637 raft_consensus.cc:1261] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:33.711216 12268 consensus_queue.cc:1035] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:33.715404 11979 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:33.718667 13069 raft_consensus.cc:2781] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Leader election won for term 2
W20240501 08:14:33.719480 8760 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.107.1:45193, user_credentials={real_user=slave}} blocked reactor thread for 58077.8us
I20240501 08:14:33.719435 8959 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:33.720742 6896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070e927b5d684367ae0b40b12a23da54" candidate_uuid: "a518ca5e9de24204bd028b0ce4a77d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:33.720985 6896 raft_consensus.cc:3037] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:33.724031 6896 raft_consensus.cc:2454] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a518ca5e9de24204bd028b0ce4a77d58 in term 2.
I20240501 08:14:33.732435 6496 catalog_manager.cc:5484] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:33.732556 6637 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:33.746708 13069 raft_consensus.cc:686] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 2 LEADER]: Becoming Leader. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Running, Role: LEADER
I20240501 08:14:33.748099 13069 consensus_queue.cc:237] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:33.792819 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet e9629fbbb4554edc9ba341d42b51f252 with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:33.799499 6497 catalog_manager.cc:5484] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 reported cstate change: term changed from 1 to 2, leader changed from 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) to a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7). New cstate: current_term: 2 leader_uuid: "a518ca5e9de24204bd028b0ce4a77d58" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:33.821213 13104 tablet_replica.cc:330] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:33.825990 13104 raft_consensus.cc:2229] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:33.826341 13104 raft_consensus.cc:2258] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:33.827260 13104 ts_tablet_manager.cc:1875] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:33.845080 13104 ts_tablet_manager.cc:1888] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:33.849315 13104 log.cc:1196] T e9629fbbb4554edc9ba341d42b51f252 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/e9629fbbb4554edc9ba341d42b51f252
I20240501 08:14:33.853415 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet e9629fbbb4554edc9ba341d42b51f252 (table rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]) successfully deleted
I20240501 08:14:33.878832 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:33.997037 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:34.003468 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:34.020959 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:34.051683 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:34.056370 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:34.132493 6768 raft_consensus.cc:1261] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:14:34.133180 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:34.134483 13069 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:34.143005 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:34.167219 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:34.177090 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:34.201295 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:34.222488 6637 raft_consensus.cc:1261] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:14:34.223232 13069 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:34.224253 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:34.241425 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:14:34.249977 12757 consensus_queue.cc:1159] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "28b7613bd4834f3ab245c82dd7e80240" responder_term: 1 status { last_received { term: 1 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 5 } }
I20240501 08:14:34.250380 6636 consensus_queue.cc:237] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:34.283433 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:34.285559 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:34.342268 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:34.354456 6899 raft_consensus.cc:1261] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:14:34.357617 8694 raft_consensus.cc:1261] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:34.357755 8563 raft_consensus.cc:1261] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:34.358466 12757 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:34.362377 13102 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:34.375300 7029 raft_consensus.cc:1261] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:34.376009 12577 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:34.383805 13069 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:34.404537 12577 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:34.410146 7028 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:34.427289 6497 catalog_manager.cc:5484] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:34.431025 8563 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:34.440354 8695 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:14:34.480625 6573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 59408.8us
I20240501 08:14:34.483207 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:34.483515 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:34.536100 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:34.539772 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:34.559911 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:34.576793 13227 tablet_replica.cc:330] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:34.578183 13227 raft_consensus.cc:2229] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:34.578995 13227 raft_consensus.cc:2258] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:34.581580 13227 ts_tablet_manager.cc:1875] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:14:34.585175 6574 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 102088us
I20240501 08:14:34.594878 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:14:34.601130 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 118217us
I20240501 08:14:34.609165 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:34.635512 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 08:14:34.636001 6571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.3:43505, user_credentials={real_user=slave}} blocked reactor thread for 153601us
I20240501 08:14:34.639719 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:34.644423 13227 ts_tablet_manager.cc:1888] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:34.650991 13227 log.cc:1196] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b9bd7500e8747c6b8ee94a1faa1c5f7
I20240501 08:14:34.652294 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7 (table rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]) successfully deleted
I20240501 08:14:34.817905 7029 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:34.854285 8827 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:34.855237 12268 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:34.868481 6768 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:34.870527 12268 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:34.875339 12268 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:34.884379 6494 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:34.882100 8827 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:34.900116 6769 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:34.909421 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ecbb30f7764fceb008baaa2c37da8c with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:34.909874 13227 tablet_replica.cc:330] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:34.910065 13227 raft_consensus.cc:2229] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:34.910436 13227 raft_consensus.cc:2258] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:34.911252 13227 ts_tablet_manager.cc:1875] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:34.917712 13227 ts_tablet_manager.cc:1888] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:34.917872 13227 log.cc:1196] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/57ecbb30f7764fceb008baaa2c37da8c
I20240501 08:14:34.929123 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 57ecbb30f7764fceb008baaa2c37da8c (table rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]) successfully deleted
I20240501 08:14:35.000669 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:35.038091 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:35.056205 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:35.057322 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:35.153390 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:35.194518 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:35.212348 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:35.219954 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:35.386933 6768 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:35.409293 8826 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:35.410295 13033 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:35.417794 8959 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:35.419143 13033 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:35.433670 6899 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:35.435122 13316 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:35.472028 13316 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:35.477043 8958 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:35.486954 6899 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:35.488365 6494 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:35.513741 8827 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:35.516139 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet a7a56c284444496b8579600a49f19d1d with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:35.518457 13365 tablet_replica.cc:330] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:35.518795 13365 raft_consensus.cc:2229] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:35.519685 13365 raft_consensus.cc:2258] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:35.520594 13365 ts_tablet_manager.cc:1875] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:35.532130 13365 ts_tablet_manager.cc:1888] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:35.532466 13365 log.cc:1196] T a7a56c284444496b8579600a49f19d1d P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/a7a56c284444496b8579600a49f19d1d
I20240501 08:14:35.533437 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet a7a56c284444496b8579600a49f19d1d (table rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]) successfully deleted
I20240501 08:14:35.610096 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:35.639708 8826 consensus_queue.cc:237] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:35.662617 6768 raft_consensus.cc:1261] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:35.664197 13182 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:35.665947 6899 raft_consensus.cc:1261] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:35.678047 13182 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:35.706080 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:35.709491 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:35.716075 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:35.717908 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:35.794607 13397 raft_consensus.cc:2932] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:35.805253 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:35.810094 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:35.812340 6489 catalog_manager.cc:5484] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 reported cstate change: config changed from index 4 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 2 leader_uuid: "a518ca5e9de24204bd028b0ce4a77d58" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:35.817601 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet 070e927b5d684367ae0b40b12a23da54 with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37350
I20240501 08:14:35.829124 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:35.834774 6768 raft_consensus.cc:2932] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:35.835834 6899 raft_consensus.cc:2932] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:35.837743 13409 tablet_replica.cc:330] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:14:35.838104 13409 raft_consensus.cc:2229] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:35.838454 13409 raft_consensus.cc:2258] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:35.839382 13409 ts_tablet_manager.cc:1875] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:35.878330 13409 ts_tablet_manager.cc:1888] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:14:35.878688 13409 log.cc:1196] T 070e927b5d684367ae0b40b12a23da54 P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/070e927b5d684367ae0b40b12a23da54
I20240501 08:14:35.880477 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet 070e927b5d684367ae0b40b12a23da54 (table rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]) successfully deleted
W20240501 08:14:35.903291 6832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.1:45193, user_credentials={real_user=slave}} blocked reactor thread for 47244.8us
I20240501 08:14:36.028045 6768 consensus_queue.cc:237] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:36.077934 8826 raft_consensus.cc:1261] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:36.079102 13033 consensus_queue.cc:1035] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:36.080533 7028 raft_consensus.cc:1261] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:36.081547 13033 consensus_queue.cc:1035] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:36.100028 13033 raft_consensus.cc:2932] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:36.104691 8826 raft_consensus.cc:2932] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:36.115574 7028 raft_consensus.cc:2932] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:36.118758 6489 catalog_manager.cc:5484] T 193de3c11ae54408a64ae3007bf88920 P a518ca5e9de24204bd028b0ce4a77d58 reported cstate change: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:36.123879 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet 193de3c11ae54408a64ae3007bf88920 with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:36.140283 13453 tablet_replica.cc:330] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:36.140554 13453 raft_consensus.cc:2229] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:36.140861 13453 raft_consensus.cc:2258] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:36.141770 13453 ts_tablet_manager.cc:1875] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:36.163833 13453 ts_tablet_manager.cc:1888] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:36.164023 13453 log.cc:1196] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/193de3c11ae54408a64ae3007bf88920
I20240501 08:14:36.165071 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 193de3c11ae54408a64ae3007bf88920 (table rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]) successfully deleted
I20240501 08:14:36.329027 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:36.358635 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:36.371848 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:36.417061 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:36.434502 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:36.527748 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:36.552263 6636 consensus_queue.cc:237] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:36.572100 7028 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:36.572846 12757 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:36.581058 8694 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:36.584156 13347 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:36.607874 8826 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:36.608711 13347 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:36.637310 12757 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:36.643620 7029 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:36.649791 6494 catalog_manager.cc:5484] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 5 to 6, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:36.656891 8694 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:36.659920 8826 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:36.662535 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:36.718348 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:36.729202 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:36.734643 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:36.765125 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:36.766422 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:36.798732 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:36.803990 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:36.835812 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:36.838191 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet dc19d80bad46460087c143778a4b0638 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:36.865888 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:36.891706 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:36.899334 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:36.914582 13582 tablet_replica.cc:330] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:36.923400 13582 raft_consensus.cc:2229] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:36.925186 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:36.932920 13582 raft_consensus.cc:2258] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:36.943687 13582 ts_tablet_manager.cc:1875] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:36.947081 6747 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:36.985375 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:36.987998 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:37.009579 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:37.044517 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:14:37.172698 6965 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 43950.4us
I20240501 08:14:37.162168 13582 ts_tablet_manager.cc:1888] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:37.196158 13582 log.cc:1196] T dc19d80bad46460087c143778a4b0638 P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc19d80bad46460087c143778a4b0638
I20240501 08:14:37.197307 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet dc19d80bad46460087c143778a4b0638 (table rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]) successfully deleted
I20240501 08:14:37.242857 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:37.259219 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:37.295830 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:37.322741 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:37.325989 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:37.356691 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:37.373804 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:37.381750 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:37.621771 7028 consensus_queue.cc:237] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:37.640863 8695 raft_consensus.cc:1261] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:37.641647 12268 consensus_queue.cc:1035] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:37.646775 6769 raft_consensus.cc:1261] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:37.647572 12268 consensus_queue.cc:1035] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:37.651281 12268 raft_consensus.cc:2932] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:37.656790 8695 raft_consensus.cc:2932] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:37.658454 6488 catalog_manager.cc:5484] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 4 to 5, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:37.677666 6768 raft_consensus.cc:2932] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:37.702589 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7f87b6032d45c5be5d748ab2cc71e2 with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37350
I20240501 08:14:37.737314 13694 tablet_replica.cc:330] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:14:37.740672 13694 raft_consensus.cc:2229] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:37.743325 13694 raft_consensus.cc:2258] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:37.746234 13694 ts_tablet_manager.cc:1875] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:37.758366 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:37.760380 7028 consensus_queue.cc:237] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:37.771114 13694 ts_tablet_manager.cc:1888] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:37.772989 13694 log.cc:1196] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/8b7f87b6032d45c5be5d748ab2cc71e2
I20240501 08:14:37.774864 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet 8b7f87b6032d45c5be5d748ab2cc71e2 (table rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]) successfully deleted
I20240501 08:14:37.803437 6769 consensus_queue.cc:237] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:37.808032 8695 raft_consensus.cc:1261] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:37.808843 13316 consensus_queue.cc:1035] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:37.813724 6637 raft_consensus.cc:1261] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:37.814463 13316 consensus_queue.cc:1035] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:37.815678 8958 raft_consensus.cc:1261] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:37.815773 8694 raft_consensus.cc:1261] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:37.816808 6636 raft_consensus.cc:1261] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:37.816882 13605 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:37.817425 12268 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:37.817878 12268 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:37.832893 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:37.857604 13316 raft_consensus.cc:2932] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:37.859282 6638 raft_consensus.cc:2932] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:37.866497 6491 catalog_manager.cc:5484] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:37.870705 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:37.871261 13605 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:37.872906 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:37.877138 8694 raft_consensus.cc:2932] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:37.877574 8695 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:37.882894 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:37.883994 6636 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:37.884727 6491 catalog_manager.cc:5484] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:37.890070 6879 tablet_service.cc:1514] Processing DeleteTablet for tablet 60238162349647c681685349c6f8ede4 with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:37.892084 8959 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:37.929829 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:37.930956 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:37.957784 13736 tablet_replica.cc:330] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:37.958050 13736 raft_consensus.cc:2229] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:37.958300 13736 raft_consensus.cc:2258] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:37.961478 13736 ts_tablet_manager.cc:1875] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:37.964633 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:37.993062 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:38.057425 13736 ts_tablet_manager.cc:1888] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:38.057637 13736 log.cc:1196] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/358b6fe9c68349cb8cb963bb3a9ed2e9
I20240501 08:14:38.058526 13736 tablet_replica.cc:330] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:38.058593 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 (table rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]) successfully deleted
I20240501 08:14:38.058862 13736 raft_consensus.cc:2229] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:38.059132 13736 raft_consensus.cc:2258] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:38.060005 13736 ts_tablet_manager.cc:1875] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:38.067131 13736 ts_tablet_manager.cc:1888] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:38.067464 13736 log.cc:1196] T 60238162349647c681685349c6f8ede4 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/60238162349647c681685349c6f8ede4
I20240501 08:14:38.068395 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 60238162349647c681685349c6f8ede4 (table rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]) successfully deleted
I20240501 08:14:38.202433 6896 consensus_queue.cc:237] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:38.209748 8695 raft_consensus.cc:1261] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:38.210058 6769 raft_consensus.cc:1261] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:38.210590 13600 consensus_queue.cc:1035] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:38.212523 13600 consensus_queue.cc:1035] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:38.230844 13600 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:38.245404 6491 catalog_manager.cc:5484] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:38.248759 6768 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:38.252511 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet 6575b94bd51546f1bd3bb862f012e7ca with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:38.262115 8695 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:38.283377 13792 tablet_replica.cc:330] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:38.283651 13792 raft_consensus.cc:2229] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:38.283948 13792 raft_consensus.cc:2258] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:38.284799 13792 ts_tablet_manager.cc:1875] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:38.296979 13792 ts_tablet_manager.cc:1888] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:38.297178 13792 log.cc:1196] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/6575b94bd51546f1bd3bb862f012e7ca
I20240501 08:14:38.298017 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet 6575b94bd51546f1bd3bb862f012e7ca (table rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]) successfully deleted
I20240501 08:14:38.479260 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:38.543797 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:38.563795 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:38.629043 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:38.660368 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:38.672866 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:38.676980 7028 consensus_queue.cc:237] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:38.745334 8959 raft_consensus.cc:1261] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:38.746081 12268 consensus_queue.cc:1035] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:38.749879 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:38.761945 12268 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:38.768463 8959 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:38.774669 6491 catalog_manager.cc:5484] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:38.781702 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:38.905874 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:38.922923 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:38.937913 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet e9629fbbb4554edc9ba341d42b51f252 with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37350
I20240501 08:14:38.939323 7028 consensus_queue.cc:237] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:38.942765 6616 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:38.951308 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:38.956327 8959 raft_consensus.cc:1261] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:38.962801 12268 consensus_queue.cc:1035] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:14:38.977104 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9629fbbb4554edc9ba341d42b51f252. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:38.979439 13854 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:38.983162 8959 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:38.987051 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:38.995795 6488 catalog_manager.cc:5484] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 6 to 7, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:39.013262 6483 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e9629fbbb4554edc9ba341d42b51f252 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:39.057909 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:39.069072 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:39.080941 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:39.113333 13916 tablet_replica.cc:330] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:14:39.113682 13916 raft_consensus.cc:2229] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:39.114147 13916 raft_consensus.cc:2258] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:39.121922 13916 ts_tablet_manager.cc:1875] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:39.132627 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:39.145159 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:39.174647 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:39.195006 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:39.219942 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:39.225525 13916 ts_tablet_manager.cc:1888] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:39.225683 13916 log.cc:1196] T e9629fbbb4554edc9ba341d42b51f252 P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/e9629fbbb4554edc9ba341d42b51f252
I20240501 08:14:39.227996 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:39.240569 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet e9629fbbb4554edc9ba341d42b51f252 (table rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]) successfully deleted
I20240501 08:14:39.287796 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:39.328339 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:39.367656 6637 consensus_queue.cc:237] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:39.408345 8694 raft_consensus.cc:1261] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:39.409144 13629 consensus_queue.cc:1035] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:39.416889 6769 raft_consensus.cc:1261] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:39.417685 13629 consensus_queue.cc:1035] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:39.438102 12757 raft_consensus.cc:2932] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:39.443192 8695 raft_consensus.cc:2932] T 37a6157d7d734a80900139944481928c P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:39.453600 6488 catalog_manager.cc:5484] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:39.460804 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:39.467007 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a6157d7d734a80900139944481928c with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:39.468962 6768 raft_consensus.cc:2932] T 37a6157d7d734a80900139944481928c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:39.479918 13982 tablet_replica.cc:330] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:39.481129 13982 raft_consensus.cc:2229] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:39.482204 13982 raft_consensus.cc:2258] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:39.483614 13982 ts_tablet_manager.cc:1875] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:39.495817 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:39.502038 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:39.531677 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:39.538580 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:39.542492 13982 ts_tablet_manager.cc:1888] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:39.544010 13982 log.cc:1196] T 37a6157d7d734a80900139944481928c P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/37a6157d7d734a80900139944481928c
I20240501 08:14:39.546326 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 37a6157d7d734a80900139944481928c (table rebalance_test_table_5 [id=016dbaa82e514f4cb4d67fdb26514c44]) successfully deleted
I20240501 08:14:39.562918 13963 ts_tablet_manager.cc:902] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:14:39.564465 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:39.600759 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:39.623149 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:39.696902 13963 tablet_copy_client.cc:308] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:14:39.697819 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet e9629fbbb4554edc9ba341d42b51f252 from peer a518ca5e9de24204bd028b0ce4a77d58 ({username='slave'} at 127.25.107.7:54455)
I20240501 08:14:39.698029 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet e9629fbbb4554edc9ba341d42b51f252 from peer a518ca5e9de24204bd028b0ce4a77d58 at {username='slave'} at 127.25.107.7:54455: session id = a518ca5e9de24204bd028b0ce4a77d58-e9629fbbb4554edc9ba341d42b51f252
I20240501 08:14:39.701277 7049 tablet_copy_source_session.cc:205] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:39.707775 13963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9629fbbb4554edc9ba341d42b51f252. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:39.718168 13963 tablet_copy_client.cc:783] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:39.718547 13963 tablet_copy_client.cc:647] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:39.721310 13963 tablet_copy_client.cc:523] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:39.725256 13963 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap starting.
I20240501 08:14:39.870422 13963 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:39.871172 13963 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58: Bootstrap complete.
I20240501 08:14:39.871578 13963 ts_tablet_manager.cc:1367] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58: Time spent bootstrapping tablet: real 0.146s user 0.057s sys 0.009s
I20240501 08:14:39.872713 13963 raft_consensus.cc:348] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:39.873142 13963 raft_consensus.cc:729] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Initialized, Role: FOLLOWER
I20240501 08:14:39.873512 13963 consensus_queue.cc:260] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:39.874400 13963 ts_tablet_manager.cc:1398] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:39.875473 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session a518ca5e9de24204bd028b0ce4a77d58-e9629fbbb4554edc9ba341d42b51f252 received from {username='slave'} at 127.25.107.7:54455
I20240501 08:14:39.875605 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session a518ca5e9de24204bd028b0ce4a77d58-e9629fbbb4554edc9ba341d42b51f252 on tablet e9629fbbb4554edc9ba341d42b51f252 with peer a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:39.896625 7028 consensus_queue.cc:237] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:39.905470 8826 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:39.905853 8561 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:39.906112 12268 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:39.906491 13854 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:39.911350 8826 raft_consensus.cc:1203] T e9629fbbb4554edc9ba341d42b51f252 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:14:39.930239 8958 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:39.932539 12268 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:39.945603 6896 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:39.947294 12268 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:39.973292 12268 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:39.976388 8959 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:40.003249 6896 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:40.012065 8825 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:40.012673 6497 catalog_manager.cc:5484] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:40.021405 8561 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:40.088176 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e70954aa4e4fcfbad12e8bfe3e0059 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:40.111537 14076 tablet_replica.cc:330] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:40.115621 14076 raft_consensus.cc:2229] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:40.117272 14076 raft_consensus.cc:2258] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:40.147013 14076 ts_tablet_manager.cc:1875] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:40.260468 14076 ts_tablet_manager.cc:1888] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:14:40.260773 14076 log.cc:1196] T a9e70954aa4e4fcfbad12e8bfe3e0059 P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/a9e70954aa4e4fcfbad12e8bfe3e0059
I20240501 08:14:40.261698 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet a9e70954aa4e4fcfbad12e8bfe3e0059 (table rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]) successfully deleted
I20240501 08:14:40.306128 7028 consensus_queue.cc:237] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:40.316991 8959 raft_consensus.cc:1261] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:40.317739 12268 consensus_queue.cc:1035] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:40.326030 6769 consensus_queue.cc:237] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:40.329811 6637 raft_consensus.cc:1261] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:40.330737 13854 consensus_queue.cc:1035] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:40.337657 12268 raft_consensus.cc:2932] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:40.342393 6636 raft_consensus.cc:2932] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:40.356052 8561 raft_consensus.cc:1261] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:40.356067 8825 raft_consensus.cc:1261] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:40.356704 6493 catalog_manager.cc:5484] T 4a6efa669c1645aaa971d59e5d38c8d5 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:40.357034 13953 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:40.357641 13953 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:40.358420 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:40.365178 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6efa669c1645aaa971d59e5d38c8d5 with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:40.367477 6896 raft_consensus.cc:1261] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:40.368347 13953 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:40.379843 8959 raft_consensus.cc:2932] T 4a6efa669c1645aaa971d59e5d38c8d5 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:40.405398 13953 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:40.409034 14127 tablet_replica.cc:330] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:40.409567 14127 raft_consensus.cc:2229] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:40.410748 14127 raft_consensus.cc:2258] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:40.411607 14127 ts_tablet_manager.cc:1875] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:40.421660 8825 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:40.421903 8561 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:40.433843 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:40.437901 6489 catalog_manager.cc:5484] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370 reported cstate change: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:40.446686 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet f93c2d0f9c7947bb83a50c7fb8c3518a with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:40.456691 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:40.469352 14140 tablet_replica.cc:330] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:40.470240 14140 raft_consensus.cc:2229] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:40.470680 14140 raft_consensus.cc:2258] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:40.471768 14140 ts_tablet_manager.cc:1875] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:40.477473 14127 ts_tablet_manager.cc:1888] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:40.477718 14127 log.cc:1196] T 4a6efa669c1645aaa971d59e5d38c8d5 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/4a6efa669c1645aaa971d59e5d38c8d5
I20240501 08:14:40.478793 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 4a6efa669c1645aaa971d59e5d38c8d5 (table rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]) successfully deleted
I20240501 08:14:40.483783 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:14:40.494458 14129 log.cc:924] Time spent T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20240501 08:14:40.497121 6896 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:40.534852 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:40.535025 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:40.540652 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:40.594424 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:40.621138 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:40.630648 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:40.641445 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:40.647389 14140 ts_tablet_manager.cc:1888] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:40.662912 6748 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:40.672029 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:40.675861 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:40.683907 14140 log.cc:1196] T f93c2d0f9c7947bb83a50c7fb8c3518a P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/f93c2d0f9c7947bb83a50c7fb8c3518a
I20240501 08:14:40.687485 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet f93c2d0f9c7947bb83a50c7fb8c3518a (table rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]) successfully deleted
I20240501 08:14:40.701930 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:40.740043 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:40.977394 7028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c"
dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47870
I20240501 08:14:40.977715 7028 raft_consensus.cc:595] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Received request to transfer leadership
W20240501 08:14:41.089092 6832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.1:45193, user_credentials={real_user=slave}} blocked reactor thread for 98087.9us
I20240501 08:14:41.137943 13854 raft_consensus.cc:979] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: : Instructing follower cd3434d87b3c46c6848a711cdf6b150c to start an election
I20240501 08:14:41.138141 13854 raft_consensus.cc:1067] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Signalling peer cd3434d87b3c46c6848a711cdf6b150c to start an election
I20240501 08:14:41.139128 6769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c"
dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
from {username='slave'} at 127.25.107.4:33789
I20240501 08:14:41.139417 6769 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.139559 6769 raft_consensus.cc:3037] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:41.142452 6769 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:41.147655 6769 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 2 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929)
I20240501 08:14:41.148556 8825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a518ca5e9de24204bd028b0ce4a77d58"
I20240501 08:14:41.148775 8825 raft_consensus.cc:3037] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:41.151674 8825 raft_consensus.cc:2454] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 2.
I20240501 08:14:41.150242 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:41.151858 7028 raft_consensus.cc:3032] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:41.151934 7028 raft_consensus.cc:729] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:14:41.152245 7028 consensus_queue.cc:260] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:41.152690 7028 raft_consensus.cc:3037] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:41.156816 6702 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a518ca5e9de24204bd028b0ce4a77d58, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:14:41.160993 13953 raft_consensus.cc:2781] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:41.171097 7028 raft_consensus.cc:2454] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 2.
I20240501 08:14:41.237008 13953 raft_consensus.cc:686] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 2 LEADER]: Becoming Leader. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:14:41.237360 13953 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:41.250586 6488 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: term changed from 1 to 2, leader changed from 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) to cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2). New cstate: current_term: 2 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:41.279929 7028 consensus_queue.cc:237] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:41.317122 8825 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:14:41.318120 12268 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:41.325889 8561 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:14:41.326722 12268 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:41.329553 8958 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:14:41.330320 12268 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:41.340214 13854 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:41.344754 8958 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:41.363610 8825 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:41.367852 6489 catalog_manager.cc:5484] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:41.385461 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e70954aa4e4fcfbad12e8bfe3e0059 with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:41.381325 8561 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:41.496996 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:41.523762 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.558686 14297 tablet_replica.cc:330] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:41.559021 14297 raft_consensus.cc:2229] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:41.559407 14297 raft_consensus.cc:2258] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:41.560407 14297 ts_tablet_manager.cc:1875] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:41.584009 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:41.589602 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.599347 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.637360 7028 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:41.644214 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.649776 8825 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:41.649818 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:41.650571 14260 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:41.667367 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:41.696457 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.705117 13953 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:41.727952 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:41.731374 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.746595 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.763942 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:41.784143 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:41.803272 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.817648 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:41.827981 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.847802 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:41.854435 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:41.860338 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:41.868230 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.880036 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:41.881532 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:41.884047 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:41.902138 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:41.912556 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:41.963194 14297 ts_tablet_manager.cc:1888] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:14:41.963338 14297 log.cc:1196] T a9e70954aa4e4fcfbad12e8bfe3e0059 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/a9e70954aa4e4fcfbad12e8bfe3e0059
I20240501 08:14:41.964193 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet a9e70954aa4e4fcfbad12e8bfe3e0059 (table rebalance_test_table_1 [id=e0b03dcc297349f0a696934f63223c7d]) successfully deleted
I20240501 08:14:41.988113 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.016332 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:42.040939 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.080861 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:42.087431 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:42.135718 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:42.723685 6637 consensus_queue.cc:237] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:42.740679 8693 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:42.741844 12757 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:42.750087 8958 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:42.753103 12757 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:42.758827 8826 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:42.759721 13977 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:42.781394 6896 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:42.782374 13977 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:42.803131 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.892714 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.907081 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.909988 13977 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:42.924718 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:42.935532 6769 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
W20240501 08:14:42.957430 6571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 53491.5us
I20240501 08:14:42.992067 8826 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:42.998349 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:43.004487 6896 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:43.009272 6489 catalog_manager.cc:5484] T 13b4e2d5fe4846e8b3c6081abb3c53da P a518ca5e9de24204bd028b0ce4a77d58 reported cstate change: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:14:43.025768 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 121226us
I20240501 08:14:43.027401 8958 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:43.050109 8826 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:43.051082 14260 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.056468 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:43.060158 6896 consensus_queue.cc:237] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:43.062026 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:43.062796 8693 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:43.073206 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:43.077250 8695 raft_consensus.cc:1261] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:43.078487 14438 consensus_queue.cc:1035] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:43.094378 14260 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:43.097463 14438 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:43.105787 8695 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:43.110884 6497 catalog_manager.cc:5484] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 5 to 6, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:43.113489 6495 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 5 to 7, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 2 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:43.115298 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet 13b4e2d5fe4846e8b3c6081abb3c53da with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:43.117105 8826 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:43.132354 6747 tablet_service.cc:1514] Processing DeleteTablet for tablet 6575b94bd51546f1bd3bb862f012e7ca with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:43.133909 6896 consensus_queue.cc:237] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:43.139883 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ecbb30f7764fceb008baaa2c37da8c with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:43.141180 6769 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
W20240501 08:14:43.145138 6702 consensus_peers.cc:485] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c -> Peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): Couldn't send request to peer e3a1c4a5b44340be8a183d5d649b27ac. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:43.145601 14521 tablet_replica.cc:330] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:43.145807 14521 raft_consensus.cc:2229] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:43.147392 14521 raft_consensus.cc:2258] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:43.148065 8826 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:43.148739 14260 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:43.151293 14521 ts_tablet_manager.cc:1875] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:43.151659 14261 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
W20240501 08:14:43.155123 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6575b94bd51546f1bd3bb862f012e7ca. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:43.155686 8693 raft_consensus.cc:1261] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:43.155794 14527 tablet_replica.cc:330] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:43.155980 14527 raft_consensus.cc:2229] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:43.156160 14527 raft_consensus.cc:2258] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:43.156169 6484 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 57ecbb30f7764fceb008baaa2c37da8c with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:43.156298 14438 consensus_queue.cc:1035] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.158985 14527 ts_tablet_manager.cc:1875] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:43.161300 8693 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:43.160017 14438 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:43.169207 6488 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New cstate: current_term: 2 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:43.170184 6495 catalog_manager.cc:5484] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 reported cstate change: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:43.172911 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6575b94bd51546f1bd3bb862f012e7ca with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:43.172850 8826 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:43.187368 14521 ts_tablet_manager.cc:1888] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:14:43.190045 14521 log.cc:1196] T 13b4e2d5fe4846e8b3c6081abb3c53da P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/13b4e2d5fe4846e8b3c6081abb3c53da
I20240501 08:14:43.191107 14527 ts_tablet_manager.cc:1888] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:14:43.191252 14527 log.cc:1196] T 57ecbb30f7764fceb008baaa2c37da8c P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/57ecbb30f7764fceb008baaa2c37da8c
I20240501 08:14:43.192046 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 13b4e2d5fe4846e8b3c6081abb3c53da (table rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]) successfully deleted
I20240501 08:14:43.192399 14521 tablet_replica.cc:330] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:43.192518 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet 57ecbb30f7764fceb008baaa2c37da8c (table rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]) successfully deleted
I20240501 08:14:43.192893 14521 raft_consensus.cc:2229] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:43.193424 14521 raft_consensus.cc:2258] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:43.194564 14521 ts_tablet_manager.cc:1875] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:43.226486 14521 ts_tablet_manager.cc:1888] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:43.231881 14521 log.cc:1196] T 6575b94bd51546f1bd3bb862f012e7ca P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/6575b94bd51546f1bd3bb862f012e7ca
I20240501 08:14:43.232897 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 6575b94bd51546f1bd3bb862f012e7ca (table rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]) successfully deleted
I20240501 08:14:43.541872 6637 consensus_queue.cc:237] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:43.554221 8958 raft_consensus.cc:1261] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:43.556519 14496 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:43.556669 6896 raft_consensus.cc:1261] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:43.557349 14496 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:43.562963 12757 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:43.563885 8959 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:43.579154 6488 catalog_manager.cc:5484] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:43.581738 6896 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:43.596594 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet 049090443eaa4b18ba90cb60af9a87ab with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:43.596949 14527 tablet_replica.cc:330] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:43.597131 14527 raft_consensus.cc:2229] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:43.597389 14527 raft_consensus.cc:2258] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:43.598148 14527 ts_tablet_manager.cc:1875] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:43.605846 14527 ts_tablet_manager.cc:1888] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:43.606001 14527 log.cc:1196] T 049090443eaa4b18ba90cb60af9a87ab P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/049090443eaa4b18ba90cb60af9a87ab
I20240501 08:14:43.607589 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet 049090443eaa4b18ba90cb60af9a87ab (table rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]) successfully deleted
I20240501 08:14:43.706522 14658 ts_tablet_manager.cc:902] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:43.732930 14668 ts_tablet_manager.cc:902] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:43.750527 14658 tablet_copy_client.cc:308] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:43.751451 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet 57ecbb30f7764fceb008baaa2c37da8c from peer e3a1c4a5b44340be8a183d5d649b27ac ({username='slave'} at 127.25.107.3:42535)
I20240501 08:14:43.751610 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet 57ecbb30f7764fceb008baaa2c37da8c from peer e3a1c4a5b44340be8a183d5d649b27ac at {username='slave'} at 127.25.107.3:42535: session id = e3a1c4a5b44340be8a183d5d649b27ac-57ecbb30f7764fceb008baaa2c37da8c
I20240501 08:14:43.754972 6789 tablet_copy_source_session.cc:205] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:43.756201 14658 ts_tablet_manager.cc:1875] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:14:43.759349 14668 tablet_copy_client.cc:308] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:14:43.761268 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 6575b94bd51546f1bd3bb862f012e7ca from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:46197)
I20240501 08:14:43.761426 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 6575b94bd51546f1bd3bb862f012e7ca from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:46197: session id = 0a814bba1abe4653bcd418a83ec42370-6575b94bd51546f1bd3bb862f012e7ca
I20240501 08:14:43.764087 6919 tablet_copy_source_session.cc:205] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:43.766510 14668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6575b94bd51546f1bd3bb862f012e7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:43.773851 14658 ts_tablet_manager.cc:1888] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 08:14:43.834417 14668 tablet_copy_client.cc:783] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:43.836885 14668 tablet_copy_client.cc:647] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:43.837594 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:43.840965 14668 tablet_copy_client.cc:523] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:43.849954 14668 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:43.866652 14658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ecbb30f7764fceb008baaa2c37da8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:43.910887 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:43.913452 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:43.925045 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:43.951215 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:43.992122 14658 tablet_copy_client.cc:783] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:43.992467 14658 tablet_copy_client.cc:647] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:43.993568 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:44.003551 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:44.008911 14658 tablet_copy_client.cc:523] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:44.012560 14658 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:14:44.018774 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:44.025617 6618 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:44.039861 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:44.075641 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:44.077759 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:44.086082 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:44.104977 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:44.115355 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:44.130426 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:44.152783 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:44.169977 14658 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:44.170425 14658 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap complete.
I20240501 08:14:44.170686 14658 ts_tablet_manager.cc:1367] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.158s user 0.046s sys 0.003s
I20240501 08:14:44.171375 14658 raft_consensus.cc:348] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:44.171557 14658 raft_consensus.cc:729] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:14:44.171757 14658 consensus_queue.cc:260] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:44.172536 14658 ts_tablet_manager.cc:1398] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:14:44.172590 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:44.188349 6896 raft_consensus.cc:1203] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 08:14:44.214416 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-57ecbb30f7764fceb008baaa2c37da8c received from {username='slave'} at 127.25.107.3:42535
I20240501 08:14:44.214570 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-57ecbb30f7764fceb008baaa2c37da8c on tablet 57ecbb30f7764fceb008baaa2c37da8c with peer e3a1c4a5b44340be8a183d5d649b27ac
I20240501 08:14:44.282647 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:44.337929 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:44.348927 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:44.383195 6748 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:44.383258 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:44.419523 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:44.734706 14668 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:44.735249 14668 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:44.735947 14668 ts_tablet_manager.cc:1367] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.886s user 0.058s sys 0.000s
I20240501 08:14:44.750586 14668 raft_consensus.cc:348] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:44.751068 14668 raft_consensus.cc:729] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:44.751470 14668 consensus_queue.cc:260] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:44.766230 14668 ts_tablet_manager.cc:1398] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20240501 08:14:44.770895 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-6575b94bd51546f1bd3bb862f012e7ca received from {username='slave'} at 127.25.107.5:46197
I20240501 08:14:44.771098 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-6575b94bd51546f1bd3bb862f012e7ca on tablet 6575b94bd51546f1bd3bb862f012e7ca with peer 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:44.784343 6637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c"
dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48376
I20240501 08:14:44.784613 6637 raft_consensus.cc:595] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 2 LEADER]: Received request to transfer leadership
I20240501 08:14:44.887555 14496 raft_consensus.cc:979] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699: : Instructing follower 28b7613bd4834f3ab245c82dd7e80240 to start an election
I20240501 08:14:44.887723 14496 raft_consensus.cc:1067] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 2 LEADER]: Signalling peer 28b7613bd4834f3ab245c82dd7e80240 to start an election
I20240501 08:14:44.890807 7027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c"
dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
from {username='slave'} at 127.25.107.1:45633
I20240501 08:14:44.891095 7027 raft_consensus.cc:482] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:44.891209 7027 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:44.896351 7027 raft_consensus.cc:504] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:44.898430 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:44.898717 6637 raft_consensus.cc:3032] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:44.898788 6637 raft_consensus.cc:729] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:44.899087 6637 consensus_queue.cc:260] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:44.899580 6637 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:44.903681 8561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "0a814bba1abe4653bcd418a83ec42370"
I20240501 08:14:44.904013 8561 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:44.908109 8693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "aba7c226aaca4e8c90931f547d48aaa3"
I20240501 08:14:44.908437 8693 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:44.918247 8693 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 3.
I20240501 08:14:44.919737 8561 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 3.
I20240501 08:14:44.920567 6965 leader_election.cc:302] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0a814bba1abe4653bcd418a83ec42370, 28b7613bd4834f3ab245c82dd7e80240, aba7c226aaca4e8c90931f547d48aaa3; no voters:
I20240501 08:14:44.921032 14750 raft_consensus.cc:2781] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:44.921167 14750 raft_consensus.cc:686] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 3 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:14:44.921487 14750 consensus_queue.cc:237] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:44.925796 6637 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 3.
I20240501 08:14:44.933226 6488 catalog_manager.cc:5484] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 2 to 3, leader changed from 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 3 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:44.933727 8825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25e1ddb987b4f8c8a4031053f581c7c" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "a518ca5e9de24204bd028b0ce4a77d58"
I20240501 08:14:44.933944 8825 raft_consensus.cc:3037] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:44.937072 8825 raft_consensus.cc:2454] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 3.
I20240501 08:14:44.937860 7027 leader_election.cc:288] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 3 election: Requested vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219), a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929)
W20240501 08:14:44.942708 8763 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 75233.3us
I20240501 08:14:45.041658 6637 consensus_queue.cc:237] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:45.052197 8959 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:45.052932 12757 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:45.059738 8561 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:45.063130 12757 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:45.109556 12757 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:45.129873 6488 catalog_manager.cc:5484] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:45.135272 6747 tablet_service.cc:1514] Processing DeleteTablet for tablet 491674dfbf74498e8a787bebaa5e95da with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:45.136673 14881 tablet_replica.cc:330] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:45.136981 14881 raft_consensus.cc:2229] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:45.137681 14881 raft_consensus.cc:2258] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:45.138585 8959 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:45.138928 14881 ts_tablet_manager.cc:1875] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:45.145664 14881 ts_tablet_manager.cc:1888] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:14:45.145890 14881 log.cc:1196] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/491674dfbf74498e8a787bebaa5e95da
I20240501 08:14:45.146919 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 491674dfbf74498e8a787bebaa5e95da (table rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]) successfully deleted
I20240501 08:14:45.152325 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:14:45.171337 6574 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 47890.1us
I20240501 08:14:45.176275 6896 consensus_queue.cc:237] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:45.178970 8561 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:45.181615 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:45.184986 8563 raft_consensus.cc:1203] T 6575b94bd51546f1bd3bb862f012e7ca P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:14:45.197547 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:45.249017 8959 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:45.249886 8561 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:45.249918 14438 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:45.251016 6769 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:45.252187 14746 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:45.252926 14746 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:45.289239 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:45.303994 14438 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:45.307714 8562 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 08:14:45.307606 8563 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:45.308305 14500 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:14:45.313308 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:45.322199 8695 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 3 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 08:14:45.325246 14500 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:14:45.335709 6497 catalog_manager.cc:5484] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:45.340549 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37350
I20240501 08:14:45.340772 6768 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:45.363716 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:45.378645 8959 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:45.404323 6637 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 08:14:45.405404 14500 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:14:45.408294 14906 tablet_replica.cc:330] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:14:45.408674 14906 raft_consensus.cc:2229] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:45.409181 14906 raft_consensus.cc:2258] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:45.421870 14906 ts_tablet_manager.cc:1875] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:45.422825 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:45.429517 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:45.460269 8825 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 08:14:45.460955 14918 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:14:45.468086 14906 ts_tablet_manager.cc:1888] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:45.482358 14906 log.cc:1196] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/399d1a392b3f4d9e8531b6e0b5ad2a61
I20240501 08:14:45.484942 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 (table rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]) successfully deleted
I20240501 08:14:45.736004 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:45.744122 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:45.763310 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:45.779588 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:45.780395 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:45.789043 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:45.798815 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:45.828294 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:45.881161 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:45.900876 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:45.930773 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:45.969432 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:46.009032 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:46.069507 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:46.098515 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:46.117888 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:46.377837 6896 consensus_queue.cc:237] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:46.381181 8693 raft_consensus.cc:1261] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:46.381693 8562 raft_consensus.cc:1261] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:46.382020 14438 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:46.382752 14438 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:46.386332 7027 raft_consensus.cc:1261] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:46.387218 15030 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:46.421195 15030 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:46.427488 7028 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:46.433135 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:46.437682 8693 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:46.446796 6489 catalog_manager.cc:5484] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:46.465039 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc0c7b429754d92b4dd166c073ac8ed with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37350
I20240501 08:14:46.473371 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:46.473394 8562 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:46.482363 15098 tablet_replica.cc:330] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:14:46.482666 15098 raft_consensus.cc:2229] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:46.485168 15098 raft_consensus.cc:2258] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:46.486622 15098 ts_tablet_manager.cc:1875] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:46.493448 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:46.524320 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:46.534725 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:46.554108 15098 ts_tablet_manager.cc:1888] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:14:46.562505 15098 log.cc:1196] T cfc0c7b429754d92b4dd166c073ac8ed P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/cfc0c7b429754d92b4dd166c073ac8ed
I20240501 08:14:46.563910 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet cfc0c7b429754d92b4dd166c073ac8ed (table rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]) successfully deleted
I20240501 08:14:46.583441 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:46.583967 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:46.609959 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:46.767323 7027 consensus_queue.cc:237] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:46.796401 8825 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 08:14:46.797345 14500 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:14:46.805249 8693 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 3 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 08:14:46.806070 14500 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:14:46.806236 8562 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 08:14:46.808701 14500 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:14:46.825946 14500 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:46.830973 8695 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:46.843708 8562 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:46.852355 6489 catalog_manager.cc:5484] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 8 to 10, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 3 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:46.857951 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet a25e1ddb987b4f8c8a4031053f581c7c with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:37350
I20240501 08:14:46.858333 15098 tablet_replica.cc:330] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:14:46.858515 15098 raft_consensus.cc:2229] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:46.858688 15098 raft_consensus.cc:2258] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:46.859433 15098 ts_tablet_manager.cc:1875] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:46.869207 8825 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:46.873636 15098 ts_tablet_manager.cc:1888] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.9
I20240501 08:14:46.879567 15098 log.cc:1196] T a25e1ddb987b4f8c8a4031053f581c7c P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/a25e1ddb987b4f8c8a4031053f581c7c
I20240501 08:14:46.880388 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet a25e1ddb987b4f8c8a4031053f581c7c (table rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]) successfully deleted
I20240501 08:14:47.001451 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:47.074959 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:47.109227 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:47.141449 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:47.192945 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:47.195633 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:47.208878 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:47.212805 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:47.238420 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:47.271041 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:47.286626 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:47.334425 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:47.374711 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:14:47.377471 6702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 77266us
W20240501 08:14:47.382655 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 102604us
I20240501 08:14:47.421356 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:47.453372 8805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:47.481336 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:47.749586 6769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c"
dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48584
I20240501 08:14:47.749830 6769 raft_consensus.cc:595] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 2 LEADER]: Received request to transfer leadership
I20240501 08:14:47.808120 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:47.884521 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:47.885097 7007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:47.920143 15104 raft_consensus.cc:979] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: : Instructing follower a518ca5e9de24204bd028b0ce4a77d58 to start an election
I20240501 08:14:47.920300 15104 raft_consensus.cc:1067] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 2 LEADER]: Signalling peer a518ca5e9de24204bd028b0ce4a77d58 to start an election
I20240501 08:14:47.922044 8826 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c"
dest_uuid: "a518ca5e9de24204bd028b0ce4a77d58"
from {username='slave'} at 127.25.107.2:37035
I20240501 08:14:47.922509 8826 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:47.922627 8826 raft_consensus.cc:3037] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:47.925417 8826 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:47.926157 8826 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [CANDIDATE]: Term 3 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:47.927966 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c" candidate_uuid: "a518ca5e9de24204bd028b0ce4a77d58" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:47.928220 6769 raft_consensus.cc:3032] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:47.928284 6769 raft_consensus.cc:729] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:14:47.928534 6769 consensus_queue.cc:260] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:47.928900 6769 raft_consensus.cc:3037] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:47.930498 6896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ecbb30f7764fceb008baaa2c37da8c" candidate_uuid: "a518ca5e9de24204bd028b0ce4a77d58" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:47.930740 6896 raft_consensus.cc:3037] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:47.933645 6896 raft_consensus.cc:2454] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a518ca5e9de24204bd028b0ce4a77d58 in term 3.
I20240501 08:14:47.938937 8760 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a518ca5e9de24204bd028b0ce4a77d58, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:47.939302 14836 raft_consensus.cc:2781] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:47.947049 6769 raft_consensus.cc:2454] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a518ca5e9de24204bd028b0ce4a77d58 in term 3.
I20240501 08:14:47.954447 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:47.955122 6747 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:47.999984 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:48.015955 6877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:48.066447 14836 raft_consensus.cc:686] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 LEADER]: Becoming Leader. State: Replica: a518ca5e9de24204bd028b0ce4a77d58, State: Running, Role: LEADER
I20240501 08:14:48.066859 14836 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:48.074076 6497 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 reported cstate change: term changed from 2 to 3, leader changed from cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) to a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7). New cstate: current_term: 3 leader_uuid: "a518ca5e9de24204bd028b0ce4a77d58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:48.078361 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:14:48.381419 6962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.1:45193, user_credentials={real_user=slave}} blocked reactor thread for 63482.3us
I20240501 08:14:48.382285 7027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f"
dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54936
I20240501 08:14:48.382508 7027 raft_consensus.cc:595] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Received request to transfer leadership
W20240501 08:14:48.395750 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 160532us
I20240501 08:14:48.400884 15389 raft_consensus.cc:979] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: : Instructing follower e3a1c4a5b44340be8a183d5d649b27ac to start an election
I20240501 08:14:48.401283 14918 raft_consensus.cc:1067] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Signalling peer e3a1c4a5b44340be8a183d5d649b27ac to start an election
I20240501 08:14:48.413138 6896 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f"
dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
from {username='slave'} at 127.25.107.4:34837
I20240501 08:14:48.413452 6896 raft_consensus.cc:482] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:48.413558 6896 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:48.417173 6896 raft_consensus.cc:504] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:48.426472 8562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "0a814bba1abe4653bcd418a83ec42370"
I20240501 08:14:48.426720 8562 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:48.430382 8562 raft_consensus.cc:2454] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 2.
I20240501 08:14:48.432632 8959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f"
I20240501 08:14:48.433060 8959 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:48.436970 8959 raft_consensus.cc:2454] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 2.
I20240501 08:14:48.437808 6833 leader_election.cc:302] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a814bba1abe4653bcd418a83ec42370, 19fdb76c8d1f41d7842e397adfc1ef3f, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:48.438360 14438 raft_consensus.cc:2781] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:48.438505 14438 raft_consensus.cc:686] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Becoming Leader. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Running, Role: LEADER
I20240501 08:14:48.438791 14438 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:48.444725 6896 leader_election.cc:288] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 2 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691), 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625)
I20240501 08:14:48.469537 7027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:48.469776 7027 raft_consensus.cc:3032] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:48.469843 7027 raft_consensus.cc:729] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:14:48.470074 7027 consensus_queue.cc:260] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:48.470481 7027 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:48.473393 7027 raft_consensus.cc:2454] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 2.
I20240501 08:14:48.473995 8561 raft_consensus.cc:1226] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Rejecting Update request from peer 28b7613bd4834f3ab245c82dd7e80240 for earlier term 1. Current term is 2. Ops: []
I20240501 08:14:48.510949 6488 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: term changed from 1 to 2, leader changed from 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) to e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3). New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:48.521852 6896 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 08:14:48.522979 14836 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:14:48.602546 6769 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 08:14:48.604456 14836 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:14:48.708531 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:48.762486 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:48.816543 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:48.827734 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:48.843196 7027 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:48.845078 14438 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:48.858510 8562 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:48.859424 14438 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:48.890681 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:48.920145 6747 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:48.931833 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:48.933744 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:48.951011 8959 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:48.953080 14438 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:48.973589 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:48.997746 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:49.004915 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:49.050565 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:49.083271 6747 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:14:49.092236 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 73195.5us
I20240501 08:14:49.130158 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:49.131366 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:49.168138 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:49.173775 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:49.232518 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:49.236328 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:49.236976 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:49.255117 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:49.317288 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:49.317700 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:49.364356 6747 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:49.378562 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:49.379072 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:49.442147 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:49.460120 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:49.503597 6747 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:49.541955 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:49.553718 7007 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:49.616761 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:49.858892 7028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8b7f87b6032d45c5be5d748ab2cc71e2"
dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54984
I20240501 08:14:49.859210 7028 raft_consensus.cc:595] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:14:49.942741 15409 raft_consensus.cc:979] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: : Instructing follower cd3434d87b3c46c6848a711cdf6b150c to start an election
I20240501 08:14:49.943033 15409 raft_consensus.cc:1067] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Signalling peer cd3434d87b3c46c6848a711cdf6b150c to start an election
I20240501 08:14:49.945605 6769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8b7f87b6032d45c5be5d748ab2cc71e2"
dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
from {username='slave'} at 127.25.107.4:33789
I20240501 08:14:49.945890 6769 raft_consensus.cc:482] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:49.945992 6769 raft_consensus.cc:3037] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:49.948792 6769 raft_consensus.cc:504] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:49.952821 8693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7f87b6032d45c5be5d748ab2cc71e2" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "aba7c226aaca4e8c90931f547d48aaa3"
I20240501 08:14:49.953043 8693 raft_consensus.cc:3037] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:49.956882 8693 raft_consensus.cc:2454] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 2.
I20240501 08:14:49.960891 6703 leader_election.cc:302] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aba7c226aaca4e8c90931f547d48aaa3, cd3434d87b3c46c6848a711cdf6b150c; no voters:
I20240501 08:14:49.961549 15470 raft_consensus.cc:2781] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:49.961953 6769 leader_election.cc:288] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 2 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
I20240501 08:14:49.971185 7028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7f87b6032d45c5be5d748ab2cc71e2" candidate_uuid: "cd3434d87b3c46c6848a711cdf6b150c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:49.971513 7028 raft_consensus.cc:3032] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:49.971580 7028 raft_consensus.cc:729] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:14:49.971875 7028 consensus_queue.cc:260] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:49.972244 7028 raft_consensus.cc:3037] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:49.975863 7028 raft_consensus.cc:2454] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3434d87b3c46c6848a711cdf6b150c in term 2.
W20240501 08:14:49.980562 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 79140.7us
I20240501 08:14:50.005158 15470 raft_consensus.cc:686] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 2 LEADER]: Becoming Leader. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:14:50.005667 15470 consensus_queue.cc:237] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:50.025235 6497 catalog_manager.cc:5484] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: term changed from 1 to 2, leader changed from 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) to cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2). New cstate: current_term: 2 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:50.057260 6636 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "049090443eaa4b18ba90cb60af9a87ab"
dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48588
I20240501 08:14:50.058118 6636 raft_consensus.cc:595] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:14:50.067700 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:50.098129 8939 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:50.128223 6618 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:14:50.130877 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:50.183482 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:50.187886 6896 consensus_queue.cc:237] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:50.193862 8959 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:50.194721 15578 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:50.194943 8561 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:50.197841 15578 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:50.198129 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:50.200578 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:50.274511 15578 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:50.279042 8957 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:50.282547 6497 catalog_manager.cc:5484] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:50.287689 8561 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:50.298549 6747 tablet_service.cc:1514] Processing DeleteTablet for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:50.312736 15502 raft_consensus.cc:979] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: : Instructing follower 19fdb76c8d1f41d7842e397adfc1ef3f to start an election
I20240501 08:14:50.312896 15502 raft_consensus.cc:1067] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Signalling peer 19fdb76c8d1f41d7842e397adfc1ef3f to start an election
I20240501 08:14:50.316550 8957 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "049090443eaa4b18ba90cb60af9a87ab"
dest_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f"
from {username='slave'} at 127.25.107.1:35209
I20240501 08:14:50.317489 8957 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:50.318008 8957 raft_consensus.cc:3037] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:50.325794 8957 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:50.345703 6636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049090443eaa4b18ba90cb60af9a87ab" candidate_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:50.345959 6636 raft_consensus.cc:3032] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:50.346019 6636 raft_consensus.cc:729] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:50.346295 6636 consensus_queue.cc:260] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:50.346596 6636 raft_consensus.cc:3037] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:50.347388 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:50.352775 6636 raft_consensus.cc:2454] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fdb76c8d1f41d7842e397adfc1ef3f in term 2.
I20240501 08:14:50.353488 8892 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f, 97cdee0bf9d84645a1451f9822c5d699; no voters:
I20240501 08:14:50.354097 15647 raft_consensus.cc:2781] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:50.357885 8825 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:50.383098 7028 raft_consensus.cc:1261] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:50.383881 15470 consensus_queue.cc:1035] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:50.386804 6896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049090443eaa4b18ba90cb60af9a87ab" candidate_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:14:50.387060 6896 raft_consensus.cc:3037] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:50.390111 6896 raft_consensus.cc:2454] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fdb76c8d1f41d7842e397adfc1ef3f in term 2.
I20240501 08:14:50.394759 8957 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 2 election: Requested vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:50.396576 15652 tablet_replica.cc:330] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:50.396965 15652 raft_consensus.cc:2229] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:50.397855 15652 raft_consensus.cc:2258] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:50.399869 15652 ts_tablet_manager.cc:1875] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:50.411250 15647 raft_consensus.cc:686] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 LEADER]: Becoming Leader. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Running, Role: LEADER
I20240501 08:14:50.412158 15647 consensus_queue.cc:237] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:50.414852 6896 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 08:14:50.415544 14836 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:14:50.442821 8693 raft_consensus.cc:1261] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:50.444912 15595 consensus_queue.cc:1035] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:50.449775 6495 catalog_manager.cc:5484] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: term changed from 1 to 2, leader changed from 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) to 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8). New cstate: current_term: 2 leader_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:50.451413 15652 ts_tablet_manager.cc:1888] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:14:50.454674 15652 log.cc:1196] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/399d1a392b3f4d9e8531b6e0b5ad2a61
I20240501 08:14:50.455966 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 (table rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]) successfully deleted
I20240501 08:14:50.503508 15681 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:50.505887 6898 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:50.515878 6495 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 8 to 10, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 3 leader_uuid: "a518ca5e9de24204bd028b0ce4a77d58" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:50.522358 6747 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ecbb30f7764fceb008baaa2c37da8c with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:50.522927 15652 tablet_replica.cc:330] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:50.523206 15652 raft_consensus.cc:2229] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:50.523459 15652 raft_consensus.cc:2258] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:50.530649 15652 ts_tablet_manager.cc:1875] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:50.554162 8825 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:50.556756 6898 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 08:14:50.557418 14836 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:14:50.562390 14836 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:50.570281 15652 ts_tablet_manager.cc:1888] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.9
I20240501 08:14:50.570442 15652 log.cc:1196] T 57ecbb30f7764fceb008baaa2c37da8c P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/57ecbb30f7764fceb008baaa2c37da8c
I20240501 08:14:50.571257 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 57ecbb30f7764fceb008baaa2c37da8c (table rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]) successfully deleted
I20240501 08:14:50.571655 6898 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:50.576889 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 57ecbb30f7764fceb008baaa2c37da8c with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:50.580003 6497 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 reported cstate change: config changed from index 10 to 11, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 3 leader_uuid: "a518ca5e9de24204bd028b0ce4a77d58" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:14:50.588537 8761 consensus_peers.cc:485] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57ecbb30f7764fceb008baaa2c37da8c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:50.820299 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:50.828872 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:50.849141 6898 raft_consensus.cc:1261] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Refusing update from remote peer 19fdb76c8d1f41d7842e397adfc1ef3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:50.850235 15647 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:50.871023 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:50.879700 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:50.943856 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:50.958007 6636 raft_consensus.cc:1261] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Refusing update from remote peer 19fdb76c8d1f41d7842e397adfc1ef3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:50.958792 15665 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:50.992632 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:51.013809 15768 ts_tablet_manager.cc:902] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929)
I20240501 08:14:51.052273 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:51.062933 15768 tablet_copy_client.cc:308] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.7:33929
I20240501 08:14:51.068620 6898 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:51.087685 8959 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:51.088172 8562 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:51.088382 14438 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:51.088820 15578 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:51.104357 15578 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:51.110853 8561 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:51.110968 8959 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:51.116861 6495 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 6 to 8, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:51.130600 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:51.132520 8847 tablet_copy_service.cc:140] P a518ca5e9de24204bd028b0ce4a77d58: Received BeginTabletCopySession request for tablet 57ecbb30f7764fceb008baaa2c37da8c from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:43617)
I20240501 08:14:51.132769 8847 tablet_copy_service.cc:161] P a518ca5e9de24204bd028b0ce4a77d58: Beginning new tablet copy session on tablet 57ecbb30f7764fceb008baaa2c37da8c from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:43617: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-57ecbb30f7764fceb008baaa2c37da8c
I20240501 08:14:51.136533 8847 tablet_copy_source_session.cc:205] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:51.138775 15768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ecbb30f7764fceb008baaa2c37da8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:51.152436 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:51.175611 15768 tablet_copy_client.cc:783] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:51.175961 15768 tablet_copy_client.cc:647] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:51.178993 15768 tablet_copy_client.cc:523] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:51.202958 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:51.203909 15768 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:51.265472 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:51.337865 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f090fda814f455a88dccd6680bdd05f with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:51.359143 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:51.367101 15824 tablet_replica.cc:330] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:51.368675 15824 raft_consensus.cc:2229] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:51.368978 15824 raft_consensus.cc:2258] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:51.370225 15824 ts_tablet_manager.cc:1875] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:51.374563 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:51.393268 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:51.395586 15824 ts_tablet_manager.cc:1888] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 08:14:51.395725 15824 log.cc:1196] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/5f090fda814f455a88dccd6680bdd05f
I20240501 08:14:51.397142 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet 5f090fda814f455a88dccd6680bdd05f (table rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]) successfully deleted
I20240501 08:14:51.401087 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:51.406514 15768 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:51.408574 15768 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:51.412074 15768 ts_tablet_manager.cc:1367] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.208s user 0.050s sys 0.004s
I20240501 08:14:51.413403 15768 raft_consensus.cc:348] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:51.413674 15768 raft_consensus.cc:729] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:51.416159 15768 consensus_queue.cc:260] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:51.416841 15768 ts_tablet_manager.cc:1398] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:14:51.419272 8847 tablet_copy_service.cc:342] P a518ca5e9de24204bd028b0ce4a77d58: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-57ecbb30f7764fceb008baaa2c37da8c received from {username='slave'} at 127.25.107.8:43617
I20240501 08:14:51.419431 8847 tablet_copy_service.cc:433] P a518ca5e9de24204bd028b0ce4a77d58: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-57ecbb30f7764fceb008baaa2c37da8c on tablet 57ecbb30f7764fceb008baaa2c37da8c with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:51.486393 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:51.817126 8959 raft_consensus.cc:1203] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11] Dedup: 3.11->[]
I20240501 08:14:51.825219 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:51.845149 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:51.849319 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:51.884634 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:51.957753 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:51.963318 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:52.084019 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:52.089562 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:52.136291 6768 consensus_queue.cc:237] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:52.186697 8695 raft_consensus.cc:1261] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:52.187533 15596 consensus_queue.cc:1035] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:52.224612 15888 raft_consensus.cc:2932] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:52.225656 8695 raft_consensus.cc:2932] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:52.242347 6489 catalog_manager.cc:5484] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 5 to 7, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 2 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:52.391947 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7f87b6032d45c5be5d748ab2cc71e2 with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:52.398514 6768 consensus_queue.cc:237] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:52.401551 8695 raft_consensus.cc:1261] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:52.402441 15888 consensus_queue.cc:1035] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:52.406811 16004 tablet_replica.cc:330] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:52.407169 16004 raft_consensus.cc:2229] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:52.407409 16004 raft_consensus.cc:2258] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:52.408263 16004 ts_tablet_manager.cc:1875] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:14:52.410923 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b7f87b6032d45c5be5d748ab2cc71e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:52.412214 15888 raft_consensus.cc:2932] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:52.419116 8695 raft_consensus.cc:2932] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:52.422236 16004 ts_tablet_manager.cc:1888] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:14:52.422657 16004 log.cc:1196] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/8b7f87b6032d45c5be5d748ab2cc71e2
I20240501 08:14:52.425187 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet 8b7f87b6032d45c5be5d748ab2cc71e2 (table rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]) successfully deleted
I20240501 08:14:52.432735 6489 catalog_manager.cc:5484] T 8b7f87b6032d45c5be5d748ab2cc71e2 P aba7c226aaca4e8c90931f547d48aaa3 reported cstate change: config changed from index 7 to 8, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 2 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:52.436615 6484 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8b7f87b6032d45c5be5d748ab2cc71e2 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:52.595820 6898 consensus_queue.cc:237] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:52.599334 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:52.622229 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:52.623497 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:52.636333 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:52.654089 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:52.657724 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:52.659993 8695 raft_consensus.cc:1261] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:52.660868 15964 consensus_queue.cc:1035] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:52.670372 6636 raft_consensus.cc:1261] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:52.671690 15964 consensus_queue.cc:1035] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:52.743264 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:52.746681 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:52.784080 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:52.807042 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:52.827493 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:52.841728 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:52.853852 15964 raft_consensus.cc:2932] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:52.858170 8695 raft_consensus.cc:2932] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:52.863468 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:52.863137 6636 raft_consensus.cc:2932] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:52.864444 6491 catalog_manager.cc:5484] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 4 to 5, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:52.868901 6747 tablet_service.cc:1514] Processing DeleteTablet for tablet 658f2a172a2d47c8bf8b44c9d51132e4 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:52.886857 16086 tablet_replica.cc:330] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:52.890372 16086 raft_consensus.cc:2229] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:52.890966 16086 raft_consensus.cc:2258] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:52.897848 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:52.902483 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:52.892366 16086 ts_tablet_manager.cc:1875] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:52.923141 16086 ts_tablet_manager.cc:1888] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:52.923444 16086 log.cc:1196] T 658f2a172a2d47c8bf8b44c9d51132e4 P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/658f2a172a2d47c8bf8b44c9d51132e4
I20240501 08:14:52.924436 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 658f2a172a2d47c8bf8b44c9d51132e4 (table rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]) successfully deleted
I20240501 08:14:52.949993 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:53.016430 16101 ts_tablet_manager.cc:902] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:14:53.167563 16101 tablet_copy_client.cc:308] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:14:53.168630 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet 8b7f87b6032d45c5be5d748ab2cc71e2 from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:41771)
I20240501 08:14:53.168809 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet 8b7f87b6032d45c5be5d748ab2cc71e2 from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:41771: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-8b7f87b6032d45c5be5d748ab2cc71e2
I20240501 08:14:53.172066 6789 tablet_copy_source_session.cc:205] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:53.173763 16101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7f87b6032d45c5be5d748ab2cc71e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:53.180846 16101 tablet_copy_client.cc:783] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:53.182104 16101 tablet_copy_client.cc:647] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:53.186282 16101 tablet_copy_client.cc:523] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:53.190315 16101 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:14:53.308017 16101 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:53.308590 16101 tablet_bootstrap.cc:492] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:14:53.308938 16101 ts_tablet_manager.cc:1367] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.119s user 0.062s sys 0.000s
I20240501 08:14:53.309773 16101 raft_consensus.cc:348] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:53.310077 16101 raft_consensus.cc:729] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:14:53.310410 16101 consensus_queue.cc:260] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:53.311093 16101 ts_tablet_manager.cc:1398] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:14:53.311975 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-8b7f87b6032d45c5be5d748ab2cc71e2 received from {username='slave'} at 127.25.107.8:41771
I20240501 08:14:53.312112 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-8b7f87b6032d45c5be5d748ab2cc71e2 on tablet 8b7f87b6032d45c5be5d748ab2cc71e2 with peer 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:53.331538 6768 consensus_queue.cc:237] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:53.338372 8561 raft_consensus.cc:1261] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:53.340049 8825 raft_consensus.cc:1261] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:53.340972 15888 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:53.341678 15888 consensus_queue.cc:1035] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:53.375057 16005 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:53.378844 8562 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P 0a814bba1abe4653bcd418a83ec42370 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:53.383641 6495 catalog_manager.cc:5484] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 6 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:53.384156 8825 raft_consensus.cc:2932] T f93c2d0f9c7947bb83a50c7fb8c3518a P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:53.408828 8959 raft_consensus.cc:1203] T 8b7f87b6032d45c5be5d748ab2cc71e2 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 08:14:53.442643 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet f93c2d0f9c7947bb83a50c7fb8c3518a with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:53.455595 16210 tablet_replica.cc:330] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:53.456307 16210 raft_consensus.cc:2229] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:53.456653 16210 raft_consensus.cc:2258] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:53.457944 16210 ts_tablet_manager.cc:1875] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:53.467525 16210 ts_tablet_manager.cc:1888] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:14:53.470842 16210 log.cc:1196] T f93c2d0f9c7947bb83a50c7fb8c3518a P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/f93c2d0f9c7947bb83a50c7fb8c3518a
I20240501 08:14:53.480785 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet f93c2d0f9c7947bb83a50c7fb8c3518a (table rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]) successfully deleted
I20240501 08:14:53.898624 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:53.965281 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:54.046293 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:54.066911 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:54.102708 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:54.106081 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:54.109283 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:54.171274 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:54.203895 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:54.207916 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:54.240783 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:54.245764 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:54.281973 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:14:54.312973 6702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 50760.2us
I20240501 08:14:54.330991 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:54.343549 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:54.346283 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:54.347117 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:54.391729 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:54.399235 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:14:54.460593 6703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 60097.9us
I20240501 08:14:54.470629 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:54.489292 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:54.502449 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:54.521767 6877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:14:54.550863 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:54.900728 7027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643"
dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42082
I20240501 08:14:54.901028 7027 raft_consensus.cc:595] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 LEADER]: Received request to transfer leadership
I20240501 08:14:54.984951 6636 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc19d80bad46460087c143778a4b0638"
dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60198
I20240501 08:14:54.985206 6636 raft_consensus.cc:595] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:14:55.073884 16337 raft_consensus.cc:979] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: : Instructing follower 28b7613bd4834f3ab245c82dd7e80240 to start an election
I20240501 08:14:55.074038 16337 raft_consensus.cc:1067] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Signalling peer 28b7613bd4834f3ab245c82dd7e80240 to start an election
I20240501 08:14:55.074924 7027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc19d80bad46460087c143778a4b0638"
dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
from {username='slave'} at 127.25.107.1:45633
I20240501 08:14:55.075131 7027 raft_consensus.cc:482] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:55.075222 7027 raft_consensus.cc:3037] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:55.078110 7027 raft_consensus.cc:504] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:55.079929 7027 leader_election.cc:288] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 2 election: Requested vote from peers 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219), a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929)
I20240501 08:14:55.084440 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:14:55.104861 6962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 45892.4us
I20240501 08:14:55.106688 6636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19d80bad46460087c143778a4b0638" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:14:55.106943 6636 raft_consensus.cc:3032] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:55.107002 6636 raft_consensus.cc:729] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:55.107267 6636 consensus_queue.cc:260] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:55.107689 6636 raft_consensus.cc:3037] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:55.112244 6636 raft_consensus.cc:2454] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 2.
I20240501 08:14:55.117722 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:55.117859 8825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19d80bad46460087c143778a4b0638" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a518ca5e9de24204bd028b0ce4a77d58"
I20240501 08:14:55.118122 8825 raft_consensus.cc:3037] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:55.121141 8825 raft_consensus.cc:2454] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 2.
I20240501 08:14:55.122997 6962 leader_election.cc:302] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699, a518ca5e9de24204bd028b0ce4a77d58; no voters:
I20240501 08:14:55.123512 16356 raft_consensus.cc:2781] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:55.123649 16356 raft_consensus.cc:686] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:14:55.123914 16356 consensus_queue.cc:237] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:55.159950 8695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc19d80bad46460087c143778a4b0638" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "aba7c226aaca4e8c90931f547d48aaa3"
I20240501 08:14:55.160238 8695 raft_consensus.cc:3037] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:55.163409 8695 raft_consensus.cc:2454] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 2.
I20240501 08:14:55.163731 6618 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:55.165285 6495 catalog_manager.cc:5484] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 1 to 2, leader changed from 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 2 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:55.186362 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:55.189476 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:55.202888 16356 raft_consensus.cc:979] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: : Instructing follower e3a1c4a5b44340be8a183d5d649b27ac to start an election
I20240501 08:14:55.203052 16356 raft_consensus.cc:1067] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 LEADER]: Signalling peer e3a1c4a5b44340be8a183d5d649b27ac to start an election
I20240501 08:14:55.203071 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:55.204025 6898 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643"
dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
from {username='slave'} at 127.25.107.4:34837
I20240501 08:14:55.204259 6898 raft_consensus.cc:482] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:55.204361 6898 raft_consensus.cc:3037] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:55.207227 6898 raft_consensus.cc:504] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:55.208585 7027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "28b7613bd4834f3ab245c82dd7e80240"
I20240501 08:14:55.208830 7027 raft_consensus.cc:3032] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:55.208907 7027 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:14:55.209185 7027 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:55.209506 7027 raft_consensus.cc:3037] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:55.209487 8561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30b4f075057e42d0bb3f8cdabdfbc643" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "0a814bba1abe4653bcd418a83ec42370"
I20240501 08:14:55.209692 8561 raft_consensus.cc:3037] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:55.212250 7027 raft_consensus.cc:2454] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 3.
I20240501 08:14:55.212741 6834 leader_election.cc:302] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:55.213089 15965 raft_consensus.cc:2781] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:55.213632 6898 leader_election.cc:288] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 election: Requested vote from peers 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:14:55.218447 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:55.229223 8561 raft_consensus.cc:2454] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 3.
I20240501 08:14:55.233778 15965 raft_consensus.cc:686] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Becoming Leader. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Running, Role: LEADER
I20240501 08:14:55.234107 15965 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:55.250317 6490 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: term changed from 2 to 3, leader changed from 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) to e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3). New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:55.259244 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:55.526803 8695 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:55.527462 16480 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:55.574847 6636 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:55.575524 16480 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:55.605109 8825 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:55.606086 16480 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:55.624173 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:55.646065 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:55.677559 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:55.678095 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:55.717249 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:55.745110 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:55.763558 8561 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 08:14:55.764242 16497 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:14:55.771432 6618 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:55.808827 6618 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:55.809512 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:55.824591 7027 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 08:14:55.825317 16497 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:14:55.830471 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:55.840091 8959 consensus_queue.cc:237] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
W20240501 08:14:55.850360 16592 log.cc:924] Time spent T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20240501 08:14:55.851496 6636 raft_consensus.cc:1261] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Refusing update from remote peer 19fdb76c8d1f41d7842e397adfc1ef3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:55.852200 15647 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:55.866994 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:55.916711 16645 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:55.918658 6636 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:55.926014 6495 catalog_manager.cc:5484] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 5 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 2 leader_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:55.927621 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:55.928025 6879 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:55.972923 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:56.004734 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:56.019061 6879 tablet_service.cc:1514] Processing DeleteTablet for tablet 049090443eaa4b18ba90cb60af9a87ab with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:56.071911 16665 tablet_replica.cc:330] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:56.072894 16665 raft_consensus.cc:2229] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:56.073632 16665 raft_consensus.cc:2258] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:56.074527 16665 ts_tablet_manager.cc:1875] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:56.113377 8959 consensus_queue.cc:237] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
W20240501 08:14:56.114837 6705 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 69794.8us
I20240501 08:14:56.118008 6636 raft_consensus.cc:1261] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Refusing update from remote peer 19fdb76c8d1f41d7842e397adfc1ef3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:56.120379 16645 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:56.124234 16645 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:56.125624 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:56.127183 6636 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:56.137754 6495 catalog_manager.cc:5484] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 7 to 8, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 2 leader_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:14:56.145959 6482 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 049090443eaa4b18ba90cb60af9a87ab with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:56.165448 16665 ts_tablet_manager.cc:1888] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:14:56.165730 16665 log.cc:1196] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/049090443eaa4b18ba90cb60af9a87ab
I20240501 08:14:56.166802 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 049090443eaa4b18ba90cb60af9a87ab (table rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]) successfully deleted
W20240501 08:14:56.194026 8893 consensus_peers.cc:485] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 049090443eaa4b18ba90cb60af9a87ab. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:56.399523 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:56.403524 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:56.421128 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:56.471578 6618 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:56.592074 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:56.603042 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:56.610916 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:56.691159 6879 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:56.725960 16765 ts_tablet_manager.cc:902] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625)
I20240501 08:14:56.807554 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:56.827438 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:56.858345 16765 tablet_copy_client.cc:308] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.8:41625
I20240501 08:14:56.862846 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:56.882131 6618 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:56.890352 6879 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:56.899287 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:56.901218 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:56.912168 8979 tablet_copy_service.cc:140] P 19fdb76c8d1f41d7842e397adfc1ef3f: Received BeginTabletCopySession request for tablet 049090443eaa4b18ba90cb60af9a87ab from peer aba7c226aaca4e8c90931f547d48aaa3 ({username='slave'} at 127.25.107.6:34707)
I20240501 08:14:56.912377 8979 tablet_copy_service.cc:161] P 19fdb76c8d1f41d7842e397adfc1ef3f: Beginning new tablet copy session on tablet 049090443eaa4b18ba90cb60af9a87ab from peer aba7c226aaca4e8c90931f547d48aaa3 at {username='slave'} at 127.25.107.6:34707: session id = aba7c226aaca4e8c90931f547d48aaa3-049090443eaa4b18ba90cb60af9a87ab
I20240501 08:14:56.917074 8979 tablet_copy_source_session.cc:205] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:56.969439 16765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049090443eaa4b18ba90cb60af9a87ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:56.975061 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:56.977124 16765 tablet_copy_client.cc:783] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:56.977537 16765 tablet_copy_client.cc:647] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:56.979847 16765 tablet_copy_client.cc:523] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:56.983639 16765 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap starting.
W20240501 08:14:57.039309 6704 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 110751us
I20240501 08:14:57.095039 7027 consensus_queue.cc:237] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:57.107878 8826 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:57.108717 16766 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:57.116204 8695 raft_consensus.cc:1261] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:57.117062 16766 consensus_queue.cc:1035] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:57.144979 16765 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:57.145524 16765 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3: Bootstrap complete.
I20240501 08:14:57.145831 16765 ts_tablet_manager.cc:1367] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3: Time spent bootstrapping tablet: real 0.162s user 0.052s sys 0.004s
I20240501 08:14:57.147310 16766 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:57.148521 16765 raft_consensus.cc:348] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:57.148710 16765 raft_consensus.cc:729] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba7c226aaca4e8c90931f547d48aaa3, State: Initialized, Role: FOLLOWER
I20240501 08:14:57.148931 16765 consensus_queue.cc:260] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:57.151573 8825 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P a518ca5e9de24204bd028b0ce4a77d58 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:57.156180 16765 ts_tablet_manager.cc:1398] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20240501 08:14:57.157259 8979 tablet_copy_service.cc:342] P 19fdb76c8d1f41d7842e397adfc1ef3f: Request end of tablet copy session aba7c226aaca4e8c90931f547d48aaa3-049090443eaa4b18ba90cb60af9a87ab received from {username='slave'} at 127.25.107.6:34707
I20240501 08:14:57.157465 8979 tablet_copy_service.cc:433] P 19fdb76c8d1f41d7842e397adfc1ef3f: ending tablet copy session aba7c226aaca4e8c90931f547d48aaa3-049090443eaa4b18ba90cb60af9a87ab on tablet 049090443eaa4b18ba90cb60af9a87ab with peer aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:57.161700 6490 catalog_manager.cc:5484] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 6 to 8, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 2 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:57.169070 8695 raft_consensus.cc:2932] T dc19d80bad46460087c143778a4b0638 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:57.187700 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet dc19d80bad46460087c143778a4b0638 with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:37350
I20240501 08:14:57.215353 16851 tablet_replica.cc:330] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:14:57.217038 16851 raft_consensus.cc:2229] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:57.217876 16851 raft_consensus.cc:2258] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:57.219352 16851 ts_tablet_manager.cc:1875] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:57.246380 16851 ts_tablet_manager.cc:1888] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 08:14:57.253717 16851 log.cc:1196] T dc19d80bad46460087c143778a4b0638 P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc19d80bad46460087c143778a4b0638
I20240501 08:14:57.254849 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet dc19d80bad46460087c143778a4b0638 (table rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]) successfully deleted
I20240501 08:14:57.271406 8826 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:14:57.297888 8957 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 08:14:57.298924 14836 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:14:57.322518 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:57.347683 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:57.384446 16884 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:57.387101 8957 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:57.389741 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:57.390817 6618 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:57.395004 6490 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 11 to 12, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 3 leader_uuid: "a518ca5e9de24204bd028b0ce4a77d58" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:14:57.399977 6879 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ecbb30f7764fceb008baaa2c37da8c with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:57.429673 16893 tablet_replica.cc:330] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:57.429991 16893 raft_consensus.cc:2229] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:57.430375 16893 raft_consensus.cc:2258] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:57.431452 16893 ts_tablet_manager.cc:1875] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:57.440671 8826 consensus_queue.cc:237] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:57.444787 8957 raft_consensus.cc:1261] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 08:14:57.445607 14836 consensus_queue.cc:1035] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:14:57.454838 16884 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:57.455397 8957 raft_consensus.cc:2932] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:57.463310 6490 catalog_manager.cc:5484] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 12 to 13, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 3 leader_uuid: "a518ca5e9de24204bd028b0ce4a77d58" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
W20240501 08:14:57.478214 8763 consensus_peers.cc:485] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57ecbb30f7764fceb008baaa2c37da8c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:57.482220 8695 raft_consensus.cc:1203] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 08:14:57.500845 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:57.526957 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 57ecbb30f7764fceb008baaa2c37da8c with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:57.532483 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:57.536429 6879 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:57.552565 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:57.565105 16893 ts_tablet_manager.cc:1888] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240501 08:14:57.565261 16893 log.cc:1196] T 57ecbb30f7764fceb008baaa2c37da8c P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/57ecbb30f7764fceb008baaa2c37da8c
I20240501 08:14:57.594895 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 57ecbb30f7764fceb008baaa2c37da8c (table rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]) successfully deleted
I20240501 08:14:57.725944 8826 consensus_queue.cc:237] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:57.733661 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:57.741014 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:57.775751 6769 raft_consensus.cc:1261] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:57.776944 16884 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:57.806669 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:57.809983 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:57.828472 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:57.829957 16884 raft_consensus.cc:2932] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:57.835022 6768 raft_consensus.cc:2932] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:57.843225 6490 catalog_manager.cc:5484] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 reported cstate change: config changed from index 6 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 2 leader_uuid: "a518ca5e9de24204bd028b0ce4a77d58" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:57.849637 8825 consensus_queue.cc:237] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:57.851075 6879 tablet_service.cc:1514] Processing DeleteTablet for tablet 070e927b5d684367ae0b40b12a23da54 with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33806
I20240501 08:14:57.851895 16893 tablet_replica.cc:330] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:14:57.852092 16893 raft_consensus.cc:2229] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:57.852288 16893 raft_consensus.cc:2258] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:57.853061 16893 ts_tablet_manager.cc:1875] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:14:57.860291 8763 consensus_peers.cc:485] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 070e927b5d684367ae0b40b12a23da54. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:57.869150 6769 raft_consensus.cc:1261] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Refusing update from remote peer a518ca5e9de24204bd028b0ce4a77d58: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:14:57.869942 16884 consensus_queue.cc:1035] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:57.872175 16893 ts_tablet_manager.cc:1888] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:14:57.873857 16893 log.cc:1196] T 070e927b5d684367ae0b40b12a23da54 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/070e927b5d684367ae0b40b12a23da54
I20240501 08:14:57.875442 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 070e927b5d684367ae0b40b12a23da54 (table rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]) successfully deleted
I20240501 08:14:57.876806 16897 raft_consensus.cc:2932] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:57.883776 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 070e927b5d684367ae0b40b12a23da54 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:57.885165 6769 raft_consensus.cc:2932] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:57.885272 6490 catalog_manager.cc:5484] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 reported cstate change: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) added. New cstate: current_term: 2 leader_uuid: "a518ca5e9de24204bd028b0ce4a77d58" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:57.894244 6618 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:57.903502 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:57.922392 6879 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:57.951023 16972 ts_tablet_manager.cc:902] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929)
I20240501 08:14:58.038770 16972 tablet_copy_client.cc:308] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.7:33929
I20240501 08:14:58.087370 8847 tablet_copy_service.cc:140] P a518ca5e9de24204bd028b0ce4a77d58: Received BeginTabletCopySession request for tablet 57ecbb30f7764fceb008baaa2c37da8c from peer 0a814bba1abe4653bcd418a83ec42370 ({username='slave'} at 127.25.107.5:35333)
I20240501 08:14:58.087630 8847 tablet_copy_service.cc:161] P a518ca5e9de24204bd028b0ce4a77d58: Beginning new tablet copy session on tablet 57ecbb30f7764fceb008baaa2c37da8c from peer 0a814bba1abe4653bcd418a83ec42370 at {username='slave'} at 127.25.107.5:35333: session id = 0a814bba1abe4653bcd418a83ec42370-57ecbb30f7764fceb008baaa2c37da8c
I20240501 08:14:58.091264 8847 tablet_copy_source_session.cc:205] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:58.095017 16972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ecbb30f7764fceb008baaa2c37da8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:58.106351 16972 tablet_copy_client.cc:783] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:58.106817 16972 tablet_copy_client.cc:647] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:58.109432 16972 tablet_copy_client.cc:523] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:58.116684 16972 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370: Bootstrap starting.
I20240501 08:14:58.145457 7027 consensus_queue.cc:237] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:14:58.169999 8825 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:58.171164 6636 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:14:58.171315 16940 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:58.172559 16944 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:58.211324 16940 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:58.218775 6637 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:58.221747 8825 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P a518ca5e9de24204bd028b0ce4a77d58 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:58.226616 6490 catalog_manager.cc:5484] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:14:58.235186 6747 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d076303ea24d7fa0fed5a9d132019c with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46892
I20240501 08:14:58.275696 17031 tablet_replica.cc:330] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:14:58.275934 17031 raft_consensus.cc:2229] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:58.276257 17031 raft_consensus.cc:2258] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:58.277209 17031 ts_tablet_manager.cc:1875] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:58.293198 17031 ts_tablet_manager.cc:1888] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:14:58.293404 17031 log.cc:1196] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/45d076303ea24d7fa0fed5a9d132019c
I20240501 08:14:58.294152 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 45d076303ea24d7fa0fed5a9d132019c (table rebalance_test_table_6 [id=56ff53c329aa409a8b7dd792d8aac79b]) successfully deleted
I20240501 08:14:58.339646 16972 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:58.341054 16972 tablet_bootstrap.cc:492] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370: Bootstrap complete.
I20240501 08:14:58.386176 17079 ts_tablet_manager.cc:902] T 070e927b5d684367ae0b40b12a23da54 P 0a814bba1abe4653bcd418a83ec42370: Initiating tablet copy from peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929)
F20240501 08:14:58.386723 17079 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-070e927b5d684367ae0b40b12a23da54.queue_time_us, tablet-download-pool-070e927b5d684367ae0b40b12a23da54.queue_time_us) (tablet-download-pool-30b4f075057e42d0bb3f8cdabdfbc643.queue_time_us, tablet-download-pool-30b4f075057e42d0bb3f8cdabdfbc643.queue_time_us) (tablet-download-pool-30b4f075057e42d0bb3f8cdabdfbc643.run_cpu_time_us, tablet-download-pool-30b4f075057e42d0bb3f8cdabdfbc643.run_cpu_time_us) (tablet-download-pool-30b4f075057e42d0bb3f8cdabdfbc643.run_wall_time_us, tablet-download-pool-30b4f075057e42d0bb3f8cdabdfbc643.run_wall_time_us) (tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.queue_time_us, tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.queue_time_us) (tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.run_cpu_time_us, tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.run_cpu_time_us) (tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.run_wall_time_us, tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.run_wall_time_us) (tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.queue_time_us, tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.queue_time_us) (tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.run_cpu_time_us, tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.run_cpu_time_us) (tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.run_wall_time_us, tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.run_wall_time_us) (tablet-download-pool-4db7f887af8f425c8fa661196add4d37.queue_time_us, tablet-download-pool-4db7f887af8f425c8fa661196add4d37.queue_time_us) (tablet-download-pool-4db7f887af8f425c8fa661196add4d37.run_cpu_time_us, tablet-download-pool-4db7f887af8f425c8fa661196add4d37.run_cpu_time_us) (tablet-download-pool-4db7f887af8f425c8fa661196add4d37.run_wall_time_us, tablet-download-pool-4db7f887af8f425c8fa661196add4d37.run_wall_time_us) (tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.queue_time_us, tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.queue_time_us) (tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_cpu_time_us, tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_cpu_time_us) (tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_wall_time_us, tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_wall_time_us) (tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.queue_time_us, tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.queue_time_us) (tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.run_cpu_time_us, tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.run_cpu_time_us) (tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.run_wall_time_us, tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.run_wall_time_us) (tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.queue_time_us, tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.queue_time_us) (tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.run_cpu_time_us, tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.run_cpu_time_us) (tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.run_wall_time_us, tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.run_wall_time_us) (tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.queue_time_us, tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.queue_time_us) (tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.run_cpu_time_us, tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.run_cpu_time_us) (tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.run_wall_time_us, tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.run_wall_time_us) (tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.queue_time_us, tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.queue_time_us) (tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_cpu_time_us, tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_cpu_time_us) (tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_wall_time_us, tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_wall_time_us) (tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.queue_time_us, tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.queue_time_us) (tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_cpu_time_us, tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_cpu_time_us) (tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_wall_time_us, tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_wall_time_us) (tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.queue_time_us, tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.queue_time_us) (tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.run_cpu_time_us, tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.run_cpu_time_us) (tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.run_wall_time_us, tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.run_wall_time_us) (tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.queue_time_us, tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.queue_time_us) (tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.run_cpu_time_us, tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.run_cpu_time_us) (tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.run_wall_time_us, tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.run_wall_time_us) (tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.queue_time_us, tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.queue_time_us) (tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.run_cpu_time_us, tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.run_cpu_time_us) (tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.run_wall_time_us, tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.run_wall_time_us) (tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.queue_time_us, tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.queue_time_us) (tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.run_cpu_time_us, tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.run_cpu_time_us) (tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.run_wall_time_us, tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f21c2dd1dcd google::LogMessage::Fail() at ??:0
@ 0x7f21c2dd5b93 google::LogMessage::SendToLog() at ??:0
I20240501 08:14:58.408911 16972 ts_tablet_manager.cc:1367] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370: Time spent bootstrapping tablet: real 0.292s user 0.064s sys 0.016s
I20240501 08:14:58.409688 16972 raft_consensus.cc:348] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:58.409889 16972 raft_consensus.cc:729] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a814bba1abe4653bcd418a83ec42370, State: Initialized, Role: FOLLOWER
I20240501 08:14:58.410102 16972 consensus_queue.cc:260] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
@ 0x7f21c2dd17cc google::LogMessage::Flush() at ??:0
@ 0x7f21c2dd2f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f21c5b12742 kudu::TraceMetrics::InternName() at ??:0
I20240501 08:14:58.438694 16972 ts_tablet_manager.cc:1398] T 57ecbb30f7764fceb008baaa2c37da8c P 0a814bba1abe4653bcd418a83ec42370: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20240501 08:14:58.440758 8847 tablet_copy_service.cc:342] P a518ca5e9de24204bd028b0ce4a77d58: Request end of tablet copy session 0a814bba1abe4653bcd418a83ec42370-57ecbb30f7764fceb008baaa2c37da8c received from {username='slave'} at 127.25.107.5:35333
I20240501 08:14:58.440907 8847 tablet_copy_service.cc:433] P a518ca5e9de24204bd028b0ce4a77d58: ending tablet copy session 0a814bba1abe4653bcd418a83ec42370-57ecbb30f7764fceb008baaa2c37da8c on tablet 57ecbb30f7764fceb008baaa2c37da8c with peer 0a814bba1abe4653bcd418a83ec42370
@ 0x7f21c5adef8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f21c5ad8cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f21d4f2b8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f21d4f43daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f21d506e480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 08:14:58.514021 6896 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:14:58.518314 8543 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:58.525401 8562 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 08:14:58.526058 15965 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:14:58.536162 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7f21d50a6a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 08:14:58.542248 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7f21d5089981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240501 08:14:58.567361 15965 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:58.570434 8561 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:58.585778 6491 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 reported cstate change: config changed from index 8 to 10, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) evicted. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:14:58.591497 7007 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with delete_type TABLET_DATA_TOMBSTONED (TS 28b7613bd4834f3ab245c82dd7e80240 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59078
I20240501 08:14:58.605968 6896 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:58.609365 17138 tablet_replica.cc:330] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: stopping tablet replica
I20240501 08:14:58.609611 17138 raft_consensus.cc:2229] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:58.609786 17138 raft_consensus.cc:2258] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:58.610543 17138 ts_tablet_manager.cc:1875] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:14:58.611219 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): Couldn't send request to peer cd3434d87b3c46c6848a711cdf6b150c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:58.611704 8561 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 08:14:58.612272 16948 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:14:58.622041 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:58.626199 16948 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:14:58.635198 8563 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:14:58.644011 6491 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P 0a814bba1abe4653bcd418a83ec42370 reported cstate change: config changed from index 10 to 11, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:14:58.657908 17138 ts_tablet_manager.cc:1888] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.9
I20240501 08:14:58.658097 17138 log.cc:1196] T 30b4f075057e42d0bb3f8cdabdfbc643 P 28b7613bd4834f3ab245c82dd7e80240: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-3/wal/wals/30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:14:58.660669 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:14:58.666318 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f21d508970e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 08:14:58.689666 6618 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:58.690315 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7f21c5ae2b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 08:14:58.720726 6879 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7f21c5ae619c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f21c5ae6019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 08:14:58.738293 6483 catalog_manager.cc:4830] TS 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): tablet 30b4f075057e42d0bb3f8cdabdfbc643 (table rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]) successfully deleted
@ 0x7f21c5ac101e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f21c8e466db start_thread at ??:0
@ 0x7f21c1c8471f clone at ??:0
I20240501 08:14:58.839325 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:58.884495 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:58.894784 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:14:58.925643 6618 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:58.947154 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:58.965118 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:59.003679 6879 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:14:59.155892 6768 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7903106d7e74e1ea102b1abdbfce63e"
dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52790
I20240501 08:14:59.156117 6768 raft_consensus.cc:595] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Received request to transfer leadership
I20240501 08:14:59.194697 17220 ts_tablet_manager.cc:902] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:14:59.290107 6618 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:59.293673 17220 tablet_copy_client.cc:308] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:14:59.294847 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer cd3434d87b3c46c6848a711cdf6b150c ({username='slave'} at 127.25.107.2:34709)
I20240501 08:14:59.295028 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer cd3434d87b3c46c6848a711cdf6b150c at {username='slave'} at 127.25.107.2:34709: session id = cd3434d87b3c46c6848a711cdf6b150c-30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:14:59.310390 6919 tablet_copy_source_session.cc:205] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:59.312763 17220 ts_tablet_manager.cc:1875] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:14:59.326514 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:14:59.349360 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:59.365980 17220 ts_tablet_manager.cc:1888] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 08:14:59.366468 17220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4f075057e42d0bb3f8cdabdfbc643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:59.366778 6747 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:14:59.373338 6703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 53518.8us
I20240501 08:14:59.389850 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:59.423985 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:14:59.446458 6574 connection.cc:535] client connection to 127.25.107.5:45691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:59.446533 6965 connection.cc:535] client connection to 127.25.107.5:45691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:59.446612 8763 connection.cc:535] client connection to 127.25.107.5:45691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:59.446735 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:59.446736 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:59.446897 8763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:59.448627 6705 connection.cc:535] client connection to 127.25.107.5:45691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:59.448933 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:59.450671 6835 connection.cc:535] client connection to 127.25.107.5:45691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:59.450968 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:59.471999 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.472255 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.487689 8763 consensus_peers.cc:462] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:14:59.494365 17220 tablet_copy_client.cc:783] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:59.494839 17220 tablet_copy_client.cc:647] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 1 WAL segments...
W20240501 08:14:59.511835 6835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 60921.1us
W20240501 08:14:59.513430 6705 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 64562.8us
W20240501 08:14:59.513815 6574 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 67133.6us
W20240501 08:14:59.522087 8763 consensus_peers.cc:462] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:14:59.540715 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.540917 6705 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.540997 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.551717 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.551883 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.551962 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.552107 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.552184 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:59.553103 6879 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:14:59.553834 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.553953 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:59.554011 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:59.556001 17220 tablet_copy_client.cc:523] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:59.563441 17260 raft_consensus.cc:979] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: : Instructing follower e3a1c4a5b44340be8a183d5d649b27ac to start an election
I20240501 08:14:59.563624 17260 raft_consensus.cc:1067] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Signalling peer e3a1c4a5b44340be8a183d5d649b27ac to start an election
I20240501 08:14:59.563868 17220 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:14:59.565901 6896 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7903106d7e74e1ea102b1abdbfce63e"
dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
from {username='slave'} at 127.25.107.2:34709
I20240501 08:14:59.566143 6896 raft_consensus.cc:482] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:59.566300 6896 raft_consensus.cc:3037] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:59.569669 6896 raft_consensus.cc:504] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:59.570535 6896 leader_election.cc:288] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 2 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:14:59.579023 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7903106d7e74e1ea102b1abdbfce63e" candidate_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:59.579299 6769 raft_consensus.cc:3032] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:59.579398 6769 raft_consensus.cc:729] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:14:59.579658 6769 consensus_queue.cc:260] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:59.579982 6769 raft_consensus.cc:3037] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:59.591069 6769 raft_consensus.cc:2454] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a1c4a5b44340be8a183d5d649b27ac in term 2.
I20240501 08:14:59.594923 6835 leader_election.cc:302] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:14:59.595276 17142 raft_consensus.cc:2781] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:59.595415 17142 raft_consensus.cc:686] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Becoming Leader. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Running, Role: LEADER
I20240501 08:14:59.595736 17142 consensus_queue.cc:237] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
W20240501 08:14:59.601100 6835 leader_election.cc:334] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:14:59.611972 6490 catalog_manager.cc:5484] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: term changed from 1 to 2, leader changed from cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) to e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3). New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } }
I20240501 08:14:59.627447 6769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "358b6fe9c68349cb8cb963bb3a9ed2e9"
dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52804
I20240501 08:14:59.627710 6769 raft_consensus.cc:595] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Received request to transfer leadership
I20240501 08:14:59.770828 17220 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:59.771311 17220 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap complete.
I20240501 08:14:59.771598 17220 ts_tablet_manager.cc:1367] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.208s user 0.056s sys 0.004s
I20240501 08:14:59.811399 17220 raft_consensus.cc:348] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:59.811715 17220 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:14:59.812002 17220 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:14:59.821092 17220 ts_tablet_manager.cc:1398] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240501 08:14:59.822842 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session cd3434d87b3c46c6848a711cdf6b150c-30b4f075057e42d0bb3f8cdabdfbc643 received from {username='slave'} at 127.25.107.2:34709
I20240501 08:14:59.823017 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session cd3434d87b3c46c6848a711cdf6b150c-30b4f075057e42d0bb3f8cdabdfbc643 on tablet 30b4f075057e42d0bb3f8cdabdfbc643 with peer cd3434d87b3c46c6848a711cdf6b150c
I20240501 08:14:59.831488 17186 raft_consensus.cc:979] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: : Instructing follower 97cdee0bf9d84645a1451f9822c5d699 to start an election
I20240501 08:14:59.831652 17186 raft_consensus.cc:1067] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Signalling peer 97cdee0bf9d84645a1451f9822c5d699 to start an election
I20240501 08:14:59.835412 6636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "358b6fe9c68349cb8cb963bb3a9ed2e9"
dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
from {username='slave'} at 127.25.107.2:37487
I20240501 08:14:59.835660 6636 raft_consensus.cc:482] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:59.835750 6636 raft_consensus.cc:3037] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:59.845526 6636 raft_consensus.cc:504] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:59.852584 8695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358b6fe9c68349cb8cb963bb3a9ed2e9" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "aba7c226aaca4e8c90931f547d48aaa3"
I20240501 08:14:59.852617 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358b6fe9c68349cb8cb963bb3a9ed2e9" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd3434d87b3c46c6848a711cdf6b150c"
I20240501 08:14:59.852896 8695 raft_consensus.cc:3037] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:59.852921 6769 raft_consensus.cc:3032] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:59.852993 6769 raft_consensus.cc:729] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Running, Role: LEADER
I20240501 08:14:59.853307 6769 consensus_queue.cc:260] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:59.853721 6769 raft_consensus.cc:3037] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:59.858443 6769 raft_consensus.cc:2454] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 2.
I20240501 08:14:59.858449 8695 raft_consensus.cc:2454] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97cdee0bf9d84645a1451f9822c5d699 in term 2.
I20240501 08:14:59.859124 6572 leader_election.cc:302] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699, aba7c226aaca4e8c90931f547d48aaa3; no voters:
I20240501 08:14:59.866436 6636 leader_election.cc:288] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 2 election: Requested vote from peers cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
I20240501 08:14:59.898245 17019 raft_consensus.cc:2781] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:59.898447 17019 raft_consensus.cc:686] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 LEADER]: Becoming Leader. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Running, Role: LEADER
I20240501 08:14:59.898738 17019 consensus_queue.cc:237] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
W20240501 08:14:59.902635 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 43556us
W20240501 08:14:59.906879 6571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 76110.1us
I20240501 08:14:59.909952 6494 catalog_manager.cc:5484] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: term changed from 1 to 2, leader changed from cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) to 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1). New cstate: current_term: 2 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:14:59.945721 8763 consensus_peers.cc:462] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:14:59.999867 6769 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:15:00.000816 17142 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:15:00.016109 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:15:00.116422 6768 raft_consensus.cc:1203] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11] Dedup: 3.11->[]
I20240501 08:15:00.142490 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:15:00.158515 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:00.160099 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:00.164803 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:00.170559 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:00.177058 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:00.178447 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:00.200139 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:00.247823 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:00.255198 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:00.261195 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:00.267247 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:15:00.273558 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:00.285148 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:00.288280 8693 raft_consensus.cc:1261] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:15:00.289237 17349 consensus_queue.cc:1035] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:15:00.303622 6768 raft_consensus.cc:1261] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:15:00.304513 17349 consensus_queue.cc:1035] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:15:00.323884 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:00.349260 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:00.356467 8763 consensus_peers.cc:462] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:00.356832 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:00.384131 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:15:00.385022 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:00.394912 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:00.429639 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:00.503966 8763 consensus_peers.cc:485] T 57ecbb30f7764fceb008baaa2c37da8c P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:00.654959 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:00.692940 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:00.700290 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:00.725096 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:15:00.726956 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:00.728122 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:00.752399 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:00.810703 8763 consensus_peers.cc:462] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:15:00.983203 6962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 61130.4us
I20240501 08:15:01.011570 6896 consensus_queue.cc:237] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:15:01.022923 8693 raft_consensus.cc:1261] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:15:01.024995 17284 consensus_queue.cc:1035] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:15:01.083106 17486 raft_consensus.cc:2932] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:15:01.084805 8695 raft_consensus.cc:2932] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:15:01.095650 6495 catalog_manager.cc:5484] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 reported cstate change: config changed from index 5 to 6, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:15:01.165755 6896 consensus_queue.cc:237] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:15:01.167850 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet 658f2a172a2d47c8bf8b44c9d51132e4 with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37350
W20240501 08:15:01.169147 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 658f2a172a2d47c8bf8b44c9d51132e4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:15:01.171109 8695 raft_consensus.cc:1261] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:15:01.171967 17486 consensus_queue.cc:1035] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:15:01.175491 17486 raft_consensus.cc:2932] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:15:01.179881 17502 tablet_replica.cc:330] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:15:01.180001 8695 raft_consensus.cc:2932] T 658f2a172a2d47c8bf8b44c9d51132e4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:15:01.181942 17502 raft_consensus.cc:2229] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:15:01.182247 17502 raft_consensus.cc:2258] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:15:01.183073 17502 ts_tablet_manager.cc:1875] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:15:01.183137 9010 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 08:14:08.588804 9013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:08.589293 9013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:08.601473 9013 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:14:11.629580 9013 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:14:13.045598 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:13.046028 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:13.046355 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:13.046581 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:13.046700 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:13.046847 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:13.046950 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:13.047104 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 314bb13099e14e298e69260ae58eb62c: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:13.047626 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:13.047763 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:13.047940 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:13.048115 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:13.048221 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:13.048333 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:13.540460 9013 rebalancer_tool.cc:1074] tablet 5f090fda814f455a88dccd6680bdd05f: '97cdee0bf9d84645a1451f9822c5d699' -> '0a814bba1abe4653bcd418a83ec42370' move scheduled
I20240501 08:14:13.890514 9013 rebalancer_tool.cc:1074] tablet 6575b94bd51546f1bd3bb862f012e7ca: '28b7613bd4834f3ab245c82dd7e80240' -> 'aba7c226aaca4e8c90931f547d48aaa3' move scheduled
I20240501 08:14:14.539037 9013 rebalancer_tool.cc:1074] tablet 13b4e2d5fe4846e8b3c6081abb3c53da: 'cd3434d87b3c46c6848a711cdf6b150c' -> 'a518ca5e9de24204bd028b0ce4a77d58' move scheduled
I20240501 08:14:14.887635 9013 rebalancer_tool.cc:1074] tablet 4a6efa669c1645aaa971d59e5d38c8d5: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move scheduled
I20240501 08:14:15.257632 9013 rebalancer_tool.cc:1074] tablet cfc0c7b429754d92b4dd166c073ac8ed: '97cdee0bf9d84645a1451f9822c5d699' -> '0a814bba1abe4653bcd418a83ec42370' move scheduled
I20240501 08:14:15.648193 9013 rebalancer_tool.cc:1084] tablet 57ecbb30f7764fceb008baaa2c37da8c: '28b7613bd4834f3ab245c82dd7e80240' -> 'aba7c226aaca4e8c90931f547d48aaa3' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 08:14:15.649859 9013 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:14:17.824802 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:17.825202 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:17.825337 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:17.825419 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:17.825507 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:17.825644 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:17.825742 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:17.825850 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:17.825956 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:17.826048 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:17.826305 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:17.826418 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:17.826541 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:17.826675 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:17.946533 9013 rebalancer_tool.cc:1074] tablet 491674dfbf74498e8a787bebaa5e95da: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:18.444953 9013 rebalancer_tool.cc:1074] tablet 193de3c11ae54408a64ae3007bf88920: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'a518ca5e9de24204bd028b0ce4a77d58' move scheduled
I20240501 08:14:18.883256 9013 rebalancer_tool.cc:1074] tablet a7a56c284444496b8579600a49f19d1d: '28b7613bd4834f3ab245c82dd7e80240' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move scheduled
I20240501 08:14:19.495995 9013 rebalancer_tool.cc:1084] tablet 358b6fe9c68349cb8cb963bb3a9ed2e9: '97cdee0bf9d84645a1451f9822c5d699' -> 'aba7c226aaca4e8c90931f547d48aaa3' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:14:19.496134 9013 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:14:21.431059 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:21.431397 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:21.431516 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:21.431680 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:21.431775 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:21.431869 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:21.431957 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:21.432056 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:21.432126 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:21.432209 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:21.618732 9013 rebalancer_tool.cc:1074] tablet e9629fbbb4554edc9ba341d42b51f252: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:22.290927 9013 rebalancer_tool.cc:1084] tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: '28b7613bd4834f3ab245c82dd7e80240' -> 'aba7c226aaca4e8c90931f547d48aaa3' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240501 08:14:22.291114 9013 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:14:23.815399 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:23.815723 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:23.816067 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:23.816156 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:23.816270 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:23.816800 9013 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:14:23.911212 9013 rebalancer_tool.cc:1074] tablet 30b4f075057e42d0bb3f8cdabdfbc643: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:24.386866 9013 rebalancer_tool.cc:1074] tablet 4db7f887af8f425c8fa661196add4d37: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:24.573982 9013 rebalancer_tool.cc:1074] tablet 78d1df7b0ea44f89b9f22c156f907c1e: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:38.319319 9013 rebalancer_tool.cc:1136] tablet a7a56c284444496b8579600a49f19d1d: '28b7613bd4834f3ab245c82dd7e80240' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move completed: OK
I20240501 08:14:38.420612 9013 rebalancer_tool.cc:1136] tablet 5f090fda814f455a88dccd6680bdd05f: '97cdee0bf9d84645a1451f9822c5d699' -> '0a814bba1abe4653bcd418a83ec42370' move completed: OK
I20240501 08:14:45.385201 9013 rebalancer_tool.cc:1136] tablet 78d1df7b0ea44f89b9f22c156f907c1e: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:14:46.928412 9013 rebalancer_tool.cc:1074] tablet 8b7f87b6032d45c5be5d748ab2cc71e2: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:47.278971 9013 rebalancer_tool.cc:1136] tablet 4db7f887af8f425c8fa661196add4d37: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
I20240501 08:14:47.557611 9013 rebalancer_tool.cc:1136] tablet 30b4f075057e42d0bb3f8cdabdfbc643: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:14:47.663909 9013 rebalancer_tool.cc:1136] tablet e9629fbbb4554edc9ba341d42b51f252: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:47.749768 9013 rebalancer_tool.cc:1136] tablet 193de3c11ae54408a64ae3007bf88920: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'a518ca5e9de24204bd028b0ce4a77d58' move completed: OK
I20240501 08:14:47.940685 9013 rebalancer_tool.cc:1136] tablet 6575b94bd51546f1bd3bb862f012e7ca: '28b7613bd4834f3ab245c82dd7e80240' -> 'aba7c226aaca4e8c90931f547d48aaa3' move completed: OK
I20240501 08:14:48.039572 9013 rebalancer_tool.cc:1136] tablet 4a6efa669c1645aaa971d59e5d38c8d5: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move completed: OK
W20240501 08:14:51.464887 15577 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 51661.5us
I20240501 08:14:52.317904 9013 rebalancer_tool.cc:1136] tablet 13b4e2d5fe4846e8b3c6081abb3c53da: 'cd3434d87b3c46c6848a711cdf6b150c' -> 'a518ca5e9de24204bd028b0ce4a77d58' move completed: OK
I20240501 08:14:52.467036 9013 rebalancer_tool.cc:1136] tablet 491674dfbf74498e8a787bebaa5e95da: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
I20240501 08:14:52.516099 9013 rebalancer_tool.cc:1136] tablet cfc0c7b429754d92b4dd166c073ac8ed: '97cdee0bf9d84645a1451f9822c5d699' -> '0a814bba1abe4653bcd418a83ec42370' move completed: OK
I20240501 08:14:52.576206 9013 rebalancer_tool.cc:1074] tablet f93c2d0f9c7947bb83a50c7fb8c3518a: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:52.859452 9013 rebalancer_tool.cc:1074] tablet 399d1a392b3f4d9e8531b6e0b5ad2a61: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:53.004685 9013 rebalancer_tool.cc:1074] tablet a25e1ddb987b4f8c8a4031053f581c7c: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:53.093060 9013 rebalancer_tool.cc:1074] tablet 57ecbb30f7764fceb008baaa2c37da8c: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:53.210073 9013 rebalancer_tool.cc:1136] tablet 57ecbb30f7764fceb008baaa2c37da8c: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:14:53.257591 9013 rebalancer_tool.cc:1136] tablet a25e1ddb987b4f8c8a4031053f581c7c: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:14:53.356189 9013 rebalancer_tool.cc:1136] tablet 399d1a392b3f4d9e8531b6e0b5ad2a61: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
I20240501 08:14:53.520809 9013 rebalancer_tool.cc:1136] tablet f93c2d0f9c7947bb83a50c7fb8c3518a: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:53.650199 9013 rebalancer_tool.cc:1136] tablet 8b7f87b6032d45c5be5d748ab2cc71e2: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:14:53.705451 9013 rebalancer_tool.cc:1074] tablet 45d076303ea24d7fa0fed5a9d132019c: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:53.749925 9013 rebalancer_tool.cc:1074] tablet 358b6fe9c68349cb8cb963bb3a9ed2e9: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:53.871472 9013 rebalancer_tool.cc:1074] tablet 29506d751571401f86bced7e6f5415d7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:53.951442 9013 rebalancer_tool.cc:1074] tablet 049090443eaa4b18ba90cb60af9a87ab: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:56.239706 9013 rebalancer_tool.cc:1136] tablet 29506d751571401f86bced7e6f5415d7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:56.294788 9013 rebalancer_tool.cc:1136] tablet 358b6fe9c68349cb8cb963bb3a9ed2e9: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:58.306473 9013 rebalancer_tool.cc:1136] tablet 049090443eaa4b18ba90cb60af9a87ab: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:58.337291 9013 rebalancer_tool.cc:1136] tablet 45d076303ea24d7fa0fed5a9d132019c: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
W20240501 08:14:59.456323 17247 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:14:59.253431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.107.5:45691 (local address 127.0.0.1:35564)
0501 08:14:59.255875 (+ 2444us) negotiation.cc:107] Waiting for socket to connect
0501 08:14:59.255898 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0501 08:14:59.256054 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:14:59.454687 (+198633us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2307,"mutex_wait_us":832,"thread_start_us":13189,"threads_started":1}
Illegal state: tablet server 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): unacceptable health status UNAVAILABLE
I20240501 08:15:01.191042 17502 ts_tablet_manager.cc:1888] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:15:01.191339 17502 log.cc:1196] T 658f2a172a2d47c8bf8b44c9d51132e4 P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/658f2a172a2d47c8bf8b44c9d51132e4
I20240501 08:15:01.192385 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet 658f2a172a2d47c8bf8b44c9d51132e4 (table rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]) successfully deleted
I20240501 08:15:01.196023 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 658f2a172a2d47c8bf8b44c9d51132e4 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:15:01.198696 6494 catalog_manager.cc:5484] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 6 to 7, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:15:01.294827 8763 consensus_peers.cc:462] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:01.314556 6637 consensus_queue.cc:237] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:15:01.319274 8695 raft_consensus.cc:1261] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:15:01.319953 17462 consensus_queue.cc:1035] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:15:01.323210 17462 raft_consensus.cc:2932] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:15:01.327252 8695 raft_consensus.cc:2932] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:15:01.334249 6494 catalog_manager.cc:5484] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 reported cstate change: config changed from index 5 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 2 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:15:01.343920 6747 tablet_service.cc:1514] Processing DeleteTablet for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46892
I20240501 08:15:01.343916 6637 consensus_queue.cc:237] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
W20240501 08:15:01.346990 6571 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 358b6fe9c68349cb8cb963bb3a9ed2e9. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:15:01.348093 8695 raft_consensus.cc:1261] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:15:01.348796 17462 consensus_queue.cc:1035] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:15:01.354409 17019 raft_consensus.cc:2932] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:15:01.358629 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:15:01.361018 6494 catalog_manager.cc:5484] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New cstate: current_term: 2 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:15:01.364660 8695 raft_consensus.cc:2932] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:15:01.370489 17541 tablet_replica.cc:330] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:15:01.370823 17541 raft_consensus.cc:2229] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:15:01.372069 17541 raft_consensus.cc:2258] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:15:01.373030 17541 ts_tablet_manager.cc:1875] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:15:01.388571 9008 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 08:14:09.020040 9016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:09.020526 9016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:09.038223 9016 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:14:12.950879 9016 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:14:14.796478 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:14.797025 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 314bb13099e14e298e69260ae58eb62c: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:14.797300 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:14.797472 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:14.797595 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:14.797827 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:14.797943 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:14.798117 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:14.798238 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:14.798383 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:14.798475 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:14.798568 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:14.798646 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 314bb13099e14e298e69260ae58eb62c: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:14.798739 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:14.798825 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:14.798923 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:14.901700 9016 rebalancer_tool.cc:1084] tablet 8b7f87b6032d45c5be5d748ab2cc71e2: '97cdee0bf9d84645a1451f9822c5d699' -> 'aba7c226aaca4e8c90931f547d48aaa3' move ignored: Illegal state: tablet 8b7f87b6032d45c5be5d748ab2cc71e2: replica aba7c226aaca4e8c90931f547d48aaa3 is already present
I20240501 08:14:14.901942 9016 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:14:16.205492 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:16.205753 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:16.207459 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:16.207823 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:16.208003 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:16.208287 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:16.208437 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:16.208535 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:16.208613 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:16.208719 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 314bb13099e14e298e69260ae58eb62c: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:16.208973 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:16.209101 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:16.209189 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:16.209355 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:16.209468 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:16.209589 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 0a814bba1abe4653bcd418a83ec42370
I20240501 08:14:16.359614 9016 rebalancer_tool.cc:1074] tablet 5f090fda814f455a88dccd6680bdd05f: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:17.263404 9016 rebalancer_tool.cc:1074] tablet 29506d751571401f86bced7e6f5415d7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'aba7c226aaca4e8c90931f547d48aaa3' move scheduled
I20240501 08:14:17.673525 9016 rebalancer_tool.cc:1074] tablet a25e1ddb987b4f8c8a4031053f581c7c: 'cd3434d87b3c46c6848a711cdf6b150c' -> '0a814bba1abe4653bcd418a83ec42370' move scheduled
I20240501 08:14:18.122534 9016 rebalancer_tool.cc:1074] tablet e9629fbbb4554edc9ba341d42b51f252: '97cdee0bf9d84645a1451f9822c5d699' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move scheduled
I20240501 08:14:18.872534 9016 rebalancer_tool.cc:1074] tablet 78d1df7b0ea44f89b9f22c156f907c1e: '28b7613bd4834f3ab245c82dd7e80240' -> 'a518ca5e9de24204bd028b0ce4a77d58' move scheduled
I20240501 08:14:19.050050 9016 rebalancer_tool.cc:1074] tablet 6575b94bd51546f1bd3bb862f012e7ca: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:19.548975 9016 rebalancer_tool.cc:1074] tablet 358b6fe9c68349cb8cb963bb3a9ed2e9: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'aba7c226aaca4e8c90931f547d48aaa3' move scheduled
I20240501 08:14:20.116437 9016 rebalancer_tool.cc:1074] tablet f93c2d0f9c7947bb83a50c7fb8c3518a: '28b7613bd4834f3ab245c82dd7e80240' -> '0a814bba1abe4653bcd418a83ec42370' move scheduled
I20240501 08:14:20.539989 9016 rebalancer_tool.cc:1074] tablet 399d1a392b3f4d9e8531b6e0b5ad2a61: '97cdee0bf9d84645a1451f9822c5d699' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move scheduled
I20240501 08:14:21.460779 9016 rebalancer_tool.cc:1074] tablet a9e70954aa4e4fcfbad12e8bfe3e0059: 'cd3434d87b3c46c6848a711cdf6b150c' -> 'a518ca5e9de24204bd028b0ce4a77d58' move scheduled
I20240501 08:14:22.063081 9016 rebalancer_tool.cc:1074] tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'aba7c226aaca4e8c90931f547d48aaa3' move scheduled
I20240501 08:14:22.419920 9016 rebalancer_tool.cc:1074] tablet a7a56c284444496b8579600a49f19d1d: '28b7613bd4834f3ab245c82dd7e80240' -> 'a518ca5e9de24204bd028b0ce4a77d58' move scheduled
I20240501 08:14:22.923300 9016 rebalancer_tool.cc:1074] tablet fdaad8e1d86942ddbc97958dcf31b344: '97cdee0bf9d84645a1451f9822c5d699' -> '0a814bba1abe4653bcd418a83ec42370' move scheduled
I20240501 08:14:23.567879 9016 rebalancer_tool.cc:1074] tablet 60238162349647c681685349c6f8ede4: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move scheduled
I20240501 08:14:23.671710 9016 rebalancer_tool.cc:1084] tablet dc19d80bad46460087c143778a4b0638: 'cd3434d87b3c46c6848a711cdf6b150c' -> 'aba7c226aaca4e8c90931f547d48aaa3' move ignored: Illegal state: tablet dc19d80bad46460087c143778a4b0638: replica aba7c226aaca4e8c90931f547d48aaa3 is already present
I20240501 08:14:23.671875 9016 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:14:25.354732 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:25.355013 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 28b7613bd4834f3ab245c82dd7e80240
W20240501 08:14:25.355121 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:25.355276 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:25.355412 9016 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:25.523368 9016 rebalancer_tool.cc:1074] tablet 491674dfbf74498e8a787bebaa5e95da: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:25.567409 9016 rebalancer_tool.cc:1074] tablet 57ecbb30f7764fceb008baaa2c37da8c: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:25.685053 9016 rebalancer_tool.cc:1074] tablet 4db7f887af8f425c8fa661196add4d37: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:28.619699 9016 rebalancer_tool.cc:1136] tablet 78d1df7b0ea44f89b9f22c156f907c1e: '28b7613bd4834f3ab245c82dd7e80240' -> 'a518ca5e9de24204bd028b0ce4a77d58' move completed: OK
I20240501 08:14:32.978852 9016 rebalancer_tool.cc:1136] tablet 5f090fda814f455a88dccd6680bdd05f: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:14:33.054824 9016 rebalancer_tool.cc:1136] tablet 29506d751571401f86bced7e6f5415d7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'aba7c226aaca4e8c90931f547d48aaa3' move completed: OK
I20240501 08:14:45.481729 9016 rebalancer_tool.cc:1136] tablet fdaad8e1d86942ddbc97958dcf31b344: '97cdee0bf9d84645a1451f9822c5d699' -> '0a814bba1abe4653bcd418a83ec42370' move completed: OK
I20240501 08:14:45.615892 9016 rebalancer_tool.cc:1136] tablet a9e70954aa4e4fcfbad12e8bfe3e0059: 'cd3434d87b3c46c6848a711cdf6b150c' -> 'a518ca5e9de24204bd028b0ce4a77d58' move completed: OK
I20240501 08:14:45.760859 9016 rebalancer_tool.cc:1136] tablet 60238162349647c681685349c6f8ede4: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move completed: OK
I20240501 08:14:45.924501 9016 rebalancer_tool.cc:1136] tablet e9629fbbb4554edc9ba341d42b51f252: '97cdee0bf9d84645a1451f9822c5d699' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move completed: OK
I20240501 08:14:46.093421 9016 rebalancer_tool.cc:1136] tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'aba7c226aaca4e8c90931f547d48aaa3' move completed: OK
I20240501 08:14:46.190773 9016 rebalancer_tool.cc:1136] tablet 491674dfbf74498e8a787bebaa5e95da: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:14:50.598095 9016 rebalancer_tool.cc:1074] tablet cfc0c7b429754d92b4dd166c073ac8ed: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:50.694604 9016 rebalancer_tool.cc:1074] tablet 193de3c11ae54408a64ae3007bf88920: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:51.044466 9016 rebalancer_tool.cc:1136] tablet 193de3c11ae54408a64ae3007bf88920: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:51.163511 9016 rebalancer_tool.cc:1136] tablet 4db7f887af8f425c8fa661196add4d37: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:51.322815 9016 rebalancer_tool.cc:1136] tablet a25e1ddb987b4f8c8a4031053f581c7c: 'cd3434d87b3c46c6848a711cdf6b150c' -> '0a814bba1abe4653bcd418a83ec42370' move completed: OK
I20240501 08:14:51.440832 9016 rebalancer_tool.cc:1136] tablet a7a56c284444496b8579600a49f19d1d: '28b7613bd4834f3ab245c82dd7e80240' -> 'a518ca5e9de24204bd028b0ce4a77d58' move completed: OK
I20240501 08:14:51.553932 9016 rebalancer_tool.cc:1136] tablet 358b6fe9c68349cb8cb963bb3a9ed2e9: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'aba7c226aaca4e8c90931f547d48aaa3' move completed: OK
I20240501 08:14:51.647996 9016 rebalancer_tool.cc:1136] tablet f93c2d0f9c7947bb83a50c7fb8c3518a: '28b7613bd4834f3ab245c82dd7e80240' -> '0a814bba1abe4653bcd418a83ec42370' move completed: OK
I20240501 08:14:51.749323 9016 rebalancer_tool.cc:1136] tablet cfc0c7b429754d92b4dd166c073ac8ed: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:14:51.916671 9016 rebalancer_tool.cc:1136] tablet 6575b94bd51546f1bd3bb862f012e7ca: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
I20240501 08:14:52.175863 9016 rebalancer_tool.cc:1136] tablet 399d1a392b3f4d9e8531b6e0b5ad2a61: '97cdee0bf9d84645a1451f9822c5d699' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move completed: OK
I20240501 08:14:52.442729 9016 rebalancer_tool.cc:1136] tablet 57ecbb30f7764fceb008baaa2c37da8c: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
I20240501 08:14:52.633908 9016 rebalancer_tool.cc:1074] tablet 049090443eaa4b18ba90cb60af9a87ab: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:52.848757 9016 rebalancer_tool.cc:1074] tablet 30b4f075057e42d0bb3f8cdabdfbc643: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:53.053754 9016 rebalancer_tool.cc:1074] tablet 658f2a172a2d47c8bf8b44c9d51132e4: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:53.109135 9016 rebalancer_tool.cc:1074] tablet 8b7f87b6032d45c5be5d748ab2cc71e2: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:53.146706 9016 rebalancer_tool.cc:1136] tablet 8b7f87b6032d45c5be5d748ab2cc71e2: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:14:53.259106 9016 rebalancer_tool.cc:1136] tablet 658f2a172a2d47c8bf8b44c9d51132e4: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
I20240501 08:14:56.409698 9016 rebalancer_tool.cc:1136] tablet 049090443eaa4b18ba90cb60af9a87ab: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:59.024327 9016 rebalancer_tool.cc:1136] tablet 30b4f075057e42d0bb3f8cdabdfbc643: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
Illegal state: tablet server 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): unacceptable health status UNAVAILABLE
I20240501 08:15:01.391654 17541 ts_tablet_manager.cc:1888] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:15:01.391834 17541 log.cc:1196] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/358b6fe9c68349cb8cb963bb3a9ed2e9
I20240501 08:15:01.392565 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 358b6fe9c68349cb8cb963bb3a9ed2e9 (table rebalance_test_table_3 [id=934448e2027044f3ae7dc3eec4d3f3dd]) successfully deleted
I20240501 08:15:01.476944 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:01.506660 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:01.518294 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:15:01.535636 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:01.545673 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:01.572675 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:01.620716 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:15:01.647585 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:01.648855 17578 ts_tablet_manager.cc:902] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
W20240501 08:15:01.654489 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:15:01.679654 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:15:01.686776 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:01.692873 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:01.725639 17578 tablet_copy_client.cc:308] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:15:01.727020 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 658f2a172a2d47c8bf8b44c9d51132e4 from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:32865)
I20240501 08:15:01.727183 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 658f2a172a2d47c8bf8b44c9d51132e4 from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:32865: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-658f2a172a2d47c8bf8b44c9d51132e4
I20240501 08:15:01.730654 6919 tablet_copy_source_session.cc:205] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:15:01.732463 17578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658f2a172a2d47c8bf8b44c9d51132e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:15:01.740322 17578 tablet_copy_client.cc:783] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 0 data blocks...
I20240501 08:15:01.740661 17578 tablet_copy_client.cc:647] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:15:01.743168 17578 tablet_copy_client.cc:523] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:15:01.748426 17578 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:15:01.750413 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:01.761727 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:01.781957 8763 consensus_peers.cc:462] T 070e927b5d684367ae0b40b12a23da54 P a518ca5e9de24204bd028b0ce4a77d58 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:01.822542 17609 ts_tablet_manager.cc:902] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P a518ca5e9de24204bd028b0ce4a77d58: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
F20240501 08:15:01.823132 17609 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-070e927b5d684367ae0b40b12a23da54.queue_time_us, tablet-download-pool-070e927b5d684367ae0b40b12a23da54.queue_time_us) (tablet-download-pool-070e927b5d684367ae0b40b12a23da54.run_cpu_time_us, tablet-download-pool-070e927b5d684367ae0b40b12a23da54.run_cpu_time_us) (tablet-download-pool-070e927b5d684367ae0b40b12a23da54.run_wall_time_us, tablet-download-pool-070e927b5d684367ae0b40b12a23da54.run_wall_time_us) (tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.queue_time_us, tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.queue_time_us) (tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_cpu_time_us, tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_cpu_time_us) (tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_wall_time_us, tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_wall_time_us) (tablet-download-pool-193de3c11ae54408a64ae3007bf88920.queue_time_us, tablet-download-pool-193de3c11ae54408a64ae3007bf88920.queue_time_us) (tablet-download-pool-193de3c11ae54408a64ae3007bf88920.run_cpu_time_us, tablet-download-pool-193de3c11ae54408a64ae3007bf88920.run_cpu_time_us) (tablet-download-pool-193de3c11ae54408a64ae3007bf88920.run_wall_time_us, tablet-download-pool-193de3c11ae54408a64ae3007bf88920.run_wall_time_us) (tablet-download-pool-29506d751571401f86bced7e6f5415d7.queue_time_us, tablet-download-pool-29506d751571401f86bced7e6f5415d7.queue_time_us) (tablet-download-pool-29506d751571401f86bced7e6f5415d7.run_cpu_time_us, tablet-download-pool-29506d751571401f86bced7e6f5415d7.run_cpu_time_us) (tablet-download-pool-29506d751571401f86bced7e6f5415d7.run_wall_time_us, tablet-download-pool-29506d751571401f86bced7e6f5415d7.run_wall_time_us) (tablet-download-pool-358b6fe9c68349cb8cb963bb3a9ed2e9.queue_time_us, tablet-download-pool-358b6fe9c68349cb8cb963bb3a9ed2e9.queue_time_us) (tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.queue_time_us, tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.queue_time_us) (tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.run_cpu_time_us, tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.run_cpu_time_us) (tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.run_wall_time_us, tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.run_wall_time_us) (tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.queue_time_us, tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.queue_time_us) (tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_cpu_time_us, tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_cpu_time_us) (tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_wall_time_us, tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_wall_time_us) (tablet-download-pool-78d1df7b0ea44f89b9f22c156f907c1e.queue_time_us, tablet-download-pool-78d1df7b0ea44f89b9f22c156f907c1e.queue_time_us) (tablet-download-pool-78d1df7b0ea44f89b9f22c156f907c1e.run_cpu_time_us, tablet-download-pool-78d1df7b0ea44f89b9f22c156f907c1e.run_cpu_time_us) (tablet-download-pool-78d1df7b0ea44f89b9f22c156f907c1e.run_wall_time_us, tablet-download-pool-78d1df7b0ea44f89b9f22c156f907c1e.run_wall_time_us) (tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.queue_time_us, tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.queue_time_us) (tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_cpu_time_us, tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_cpu_time_us) (tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_wall_time_us, tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_wall_time_us) (tablet-download-pool-a7a56c284444496b8579600a49f19d1d.queue_time_us, tablet-download-pool-a7a56c284444496b8579600a49f19d1d.queue_time_us) (tablet-download-pool-a7a56c284444496b8579600a49f19d1d.run_cpu_time_us, tablet-download-pool-a7a56c284444496b8579600a49f19d1d.run_cpu_time_us) (tablet-download-pool-a7a56c284444496b8579600a49f19d1d.run_wall_time_us, tablet-download-pool-a7a56c284444496b8579600a49f19d1d.run_wall_time_us) (tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.queue_time_us, tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.queue_time_us) (tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_cpu_time_us, tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_cpu_time_us) (tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_wall_time_us, tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_wall_time_us) (tablet-download-pool-dc19d80bad46460087c143778a4b0638.queue_time_us, tablet-download-pool-dc19d80bad46460087c143778a4b0638.queue_time_us) (tablet-download-pool-dc19d80bad46460087c143778a4b0638.run_cpu_time_us, tablet-download-pool-dc19d80bad46460087c143778a4b0638.run_cpu_time_us) (tablet-download-pool-dc19d80bad46460087c143778a4b0638.run_wall_time_us, tablet-download-pool-dc19d80bad46460087c143778a4b0638.run_wall_time_us) (tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.queue_time_us, tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.queue_time_us) (tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.run_cpu_time_us, tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.run_cpu_time_us) (tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.run_wall_time_us, tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.run_wall_time_us) (tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.queue_time_us, tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.queue_time_us) (tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.run_cpu_time_us, tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.run_cpu_time_us) (tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.run_wall_time_us, tablet-download-pool-f93c2d0f9c7947bb83a50c7fb8c3518a.run_wall_time_us) (tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.queue_time_us, tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.queue_time_us) (tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.run_cpu_time_us, tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.run_cpu_time_us) (tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.run_wall_time_us, tablet-download-pool-fdaad8e1d86942ddbc97958dcf31b344.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 08:15:01.829846 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f2ed0f32dcd google::LogMessage::Fail() at ??:0
I20240501 08:15:01.839993 8805 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:15:01.842837 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
@ 0x7f2ed0f36b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f2ed0f327cc google::LogMessage::Flush() at ??:0
@ 0x7f2ed0f33f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
W20240501 08:15:01.877554 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:01.879735 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7f2ed3c73742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f2ed3c3ff8b kudu::ThreadPool::ThreadPool() at ??:0
I20240501 08:15:01.933554 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7f2ed3c39cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 08:15:01.945008 17578 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:15:01.945385 17578 tablet_bootstrap.cc:492] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:15:01.945617 17578 ts_tablet_manager.cc:1367] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.197s user 0.048s sys 0.008s
W20240501 08:15:01.946020 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:01.946326 17578 raft_consensus.cc:348] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:15:01.946525 17578 raft_consensus.cc:729] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:15:01.946745 17578 consensus_queue.cc:260] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:15:01.947227 17578 ts_tablet_manager.cc:1398] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240501 08:15:01.947855 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-658f2a172a2d47c8bf8b44c9d51132e4 received from {username='slave'} at 127.25.107.8:32865
I20240501 08:15:01.947957 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-658f2a172a2d47c8bf8b44c9d51132e4 on tablet 658f2a172a2d47c8bf8b44c9d51132e4 with peer 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:15:01.948632 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:15:01.949568 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:01.955332 8959 raft_consensus.cc:1203] T 658f2a172a2d47c8bf8b44c9d51132e4 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
@ 0x7f2ee308c8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f2ee30a4daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f2ee31cf480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f2ee3207a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f2ee31ea981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f2ee31ea70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f2ed3c43b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2ed3c4719c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2ed3c47019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f2ed3c2201e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2ed6fa76db start_thread at ??:0
@ 0x7f2ecfde571f clone at ??:0
I20240501 08:15:02.120999 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:02.135497 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:02.136441 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:02.151165 8959 consensus_queue.cc:237] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
W20240501 08:15:02.163487 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:15:02.165153 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:02.191977 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:02.192867 8695 raft_consensus.cc:1261] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer 19fdb76c8d1f41d7842e397adfc1ef3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:15:02.193500 16755 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:15:02.196787 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:02.202836 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:02.223817 16755 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:15:02.231501 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:02.258613 8695 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:15:02.267417 6490 catalog_manager.cc:5484] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 8 to 9, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 2 leader_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:15:02.300773 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet 049090443eaa4b18ba90cb60af9a87ab with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37350
I20240501 08:15:02.301096 8959 consensus_queue.cc:237] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:15:02.301419 17678 tablet_replica.cc:330] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:15:02.301599 17678 raft_consensus.cc:2229] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:15:02.301813 17678 raft_consensus.cc:2258] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240501 08:15:02.303052 8892 consensus_peers.cc:485] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f -> Peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): Couldn't send request to peer e3a1c4a5b44340be8a183d5d649b27ac. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:15:02.303364 8695 raft_consensus.cc:1261] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Refusing update from remote peer 19fdb76c8d1f41d7842e397adfc1ef3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:15:02.303839 16755 consensus_queue.cc:1035] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:15:02.305231 17678 ts_tablet_manager.cc:1875] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:15:02.305999 16755 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:15:02.306423 8695 raft_consensus.cc:2932] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:15:02.312031 6495 catalog_manager.cc:5484] T 049090443eaa4b18ba90cb60af9a87ab P aba7c226aaca4e8c90931f547d48aaa3 reported cstate change: config changed from index 9 to 10, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New cstate: current_term: 2 leader_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:15:02.314190 6482 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 049090443eaa4b18ba90cb60af9a87ab with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:15:02.324975 17678 ts_tablet_manager.cc:1888] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 08:15:02.325110 17678 log.cc:1196] T 049090443eaa4b18ba90cb60af9a87ab P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/049090443eaa4b18ba90cb60af9a87ab
I20240501 08:15:02.325966 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet 049090443eaa4b18ba90cb60af9a87ab (table rebalance_test_table_9 [id=46df7d4f86e644a0818048a28715a3d1]) successfully deleted
W20240501 08:15:02.400369 6962 connection.cc:535] client connection to 127.25.107.7:33929 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:15:02.400637 6702 connection.cc:535] client connection to 127.25.107.7:33929 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:15:02.400867 6571 connection.cc:535] client connection to 127.25.107.7:33929 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:15:02.403477 17624 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:15:02.401477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.107.7:33929 (local address 127.25.107.4:50245)
0501 08:15:02.401616 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 08:15:02.401634 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0501 08:15:02.401746 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:15:02.402815 (+ 1069us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":38}
W20240501 08:15:02.403477 17277 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:15:02.402125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.107.7:33929 (local address 127.25.107.2:57819)
0501 08:15:02.402292 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0501 08:15:02.402309 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 08:15:02.402411 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:15:02.402683 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
W20240501 08:15:02.403662 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:15:02.403765 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:15:02.403837 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:15:02.404465 17278 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:15:02.401861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.107.7:33929 (local address 127.25.107.1:48945)
0501 08:15:02.402161 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0501 08:15:02.402177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 08:15:02.402434 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:15:02.403837 (+ 1403us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":198}
W20240501 08:15:02.404807 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:15:02.404928 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:15:02.405019 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:15:02.409034 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:15:02.409157 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:15:02.409220 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:15:02.421329 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:15:02.511544 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:02.516072 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:15:02.517971 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:02.546187 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:02.565214 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:15:02.568130 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:15:02.573562 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:02.590014 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:02.590201 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:02.646627 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:02.725315 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:02.753518 17773 ts_tablet_manager.cc:902] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Initiating tablet copy from peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625)
I20240501 08:15:02.794562 17773 tablet_copy_client.cc:308] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.8:41625
I20240501 08:15:02.795383 8979 tablet_copy_service.cc:140] P 19fdb76c8d1f41d7842e397adfc1ef3f: Received BeginTabletCopySession request for tablet 049090443eaa4b18ba90cb60af9a87ab from peer e3a1c4a5b44340be8a183d5d649b27ac ({username='slave'} at 127.25.107.3:47407)
I20240501 08:15:02.795534 8979 tablet_copy_service.cc:161] P 19fdb76c8d1f41d7842e397adfc1ef3f: Beginning new tablet copy session on tablet 049090443eaa4b18ba90cb60af9a87ab from peer e3a1c4a5b44340be8a183d5d649b27ac at {username='slave'} at 127.25.107.3:47407: session id = e3a1c4a5b44340be8a183d5d649b27ac-049090443eaa4b18ba90cb60af9a87ab
I20240501 08:15:02.798245 8979 tablet_copy_source_session.cc:205] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:15:02.801010 17773 ts_tablet_manager.cc:1875] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:15:02.807219 17773 ts_tablet_manager.cc:1888] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 08:15:02.808559 17773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049090443eaa4b18ba90cb60af9a87ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:15:02.812034 17773 tablet_copy_client.cc:783] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 0 data blocks...
I20240501 08:15:02.812397 17773 tablet_copy_client.cc:647] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 1 WAL segments...
I20240501 08:15:02.814050 17773 tablet_copy_client.cc:523] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:15:02.817588 17773 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:15:02.864837 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:02.888834 17773 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:15:02.889417 17773 tablet_bootstrap.cc:492] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap complete.
I20240501 08:15:02.889734 17773 ts_tablet_manager.cc:1367] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.072s user 0.047s sys 0.007s
I20240501 08:15:02.890630 17773 raft_consensus.cc:348] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:15:02.890856 17773 raft_consensus.cc:729] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:15:02.891060 17773 consensus_queue.cc:260] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:15:02.891660 17773 ts_tablet_manager.cc:1398] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:15:02.892630 8979 tablet_copy_service.cc:342] P 19fdb76c8d1f41d7842e397adfc1ef3f: Request end of tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-049090443eaa4b18ba90cb60af9a87ab received from {username='slave'} at 127.25.107.3:47407
I20240501 08:15:02.892741 8979 tablet_copy_service.cc:433] P 19fdb76c8d1f41d7842e397adfc1ef3f: ending tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-049090443eaa4b18ba90cb60af9a87ab on tablet 049090443eaa4b18ba90cb60af9a87ab with peer e3a1c4a5b44340be8a183d5d649b27ac
I20240501 08:15:02.905057 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:02.935547 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:02.956360 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:02.995611 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:03.019829 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:03.033190 7027 consensus_queue.cc:237] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:15:03.059443 8959 raft_consensus.cc:1261] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:15:03.060348 17238 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:15:03.064577 8695 raft_consensus.cc:1261] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:15:03.065405 17749 consensus_queue.cc:1035] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:15:03.075100 17238 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:15:03.077636 8957 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:15:03.083896 6493 catalog_manager.cc:5484] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 6 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:15:03.129700 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:03.134328 8693 raft_consensus.cc:2932] T 60238162349647c681685349c6f8ede4 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:15:03.152216 6618 tablet_service.cc:1514] Processing DeleteTablet for tablet 60238162349647c681685349c6f8ede4 with delete_type TABLET_DATA_TOMBSTONED (TS 97cdee0bf9d84645a1451f9822c5d699 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37350
I20240501 08:15:03.152982 17839 tablet_replica.cc:330] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699: stopping tablet replica
I20240501 08:15:03.154002 17839 raft_consensus.cc:2229] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:15:03.156046 17839 raft_consensus.cc:2258] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:15:03.156888 17839 ts_tablet_manager.cc:1875] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:15:03.175500 16755 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:03.175685 16755 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:03.176640 16755 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:15:03.190100 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:03.201198 17839 ts_tablet_manager.cc:1888] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:15:03.202713 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:03.208681 17839 log.cc:1196] T 60238162349647c681685349c6f8ede4 P 97cdee0bf9d84645a1451f9822c5d699: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-0/wal/wals/60238162349647c681685349c6f8ede4
I20240501 08:15:03.210656 6481 catalog_manager.cc:4830] TS 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): tablet 60238162349647c681685349c6f8ede4 (table rebalance_test_table_7 [id=38f32905d7f144068b86f5b2602e5703]) successfully deleted
I20240501 08:15:03.229293 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:03.233052 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:03.259370 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:03.270629 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:03.272382 6896 raft_consensus.cc:1203] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20240501 08:15:03.273536 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:03.273662 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:03.273885 16755 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:15:03.277222 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:03.293380 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:03.390483 17580 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:03.390692 17580 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:03.391404 17580 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:03.393620 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:15:03.402114 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:03.402285 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:03.402549 17580 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:15:03.648324 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:03.754617 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:03.774900 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:03.845315 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:03.848717 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:03.858994 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:03.906477 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:03.938872 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:03.948405 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:03.950600 6833 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 51120.2us
I20240501 08:15:03.956331 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:03.962508 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:04.003245 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:04.020761 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:15:04.030838 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:04.056609 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:04.063457 9009 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 08:14:08.654129 9014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:08.654583 9014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:08.667738 9014 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:14:11.644575 9014 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:14:13.655634 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:13.656500 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:13.656870 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:13.657002 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:13.657116 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:13.658108 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:13.658288 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 314bb13099e14e298e69260ae58eb62c: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:13.658416 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:13.658497 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:13.658609 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:13.658757 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:13.658862 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:13.658972 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:13.659106 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:14.444185 9014 rebalancer_tool.cc:1074] tablet 8b7f87b6032d45c5be5d748ab2cc71e2: '97cdee0bf9d84645a1451f9822c5d699' -> 'aba7c226aaca4e8c90931f547d48aaa3' move scheduled
I20240501 08:14:14.754024 9014 rebalancer_tool.cc:1074] tablet 491674dfbf74498e8a787bebaa5e95da: '28b7613bd4834f3ab245c82dd7e80240' -> '0a814bba1abe4653bcd418a83ec42370' move scheduled
I20240501 08:14:15.040593 9014 rebalancer_tool.cc:1074] tablet 13b4e2d5fe4846e8b3c6081abb3c53da: 'cd3434d87b3c46c6848a711cdf6b150c' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move scheduled
I20240501 08:14:15.437745 9014 rebalancer_tool.cc:1074] tablet 57ecbb30f7764fceb008baaa2c37da8c: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'a518ca5e9de24204bd028b0ce4a77d58' move scheduled
I20240501 08:14:15.893152 9014 rebalancer_tool.cc:1074] tablet cfc0c7b429754d92b4dd166c073ac8ed: '97cdee0bf9d84645a1451f9822c5d699' -> 'aba7c226aaca4e8c90931f547d48aaa3' move scheduled
I20240501 08:14:16.634119 9014 rebalancer_tool.cc:1074] tablet d7903106d7e74e1ea102b1abdbfce63e: '28b7613bd4834f3ab245c82dd7e80240' -> '0a814bba1abe4653bcd418a83ec42370' move scheduled
I20240501 08:14:17.338271 9014 rebalancer_tool.cc:1074] tablet a9e70954aa4e4fcfbad12e8bfe3e0059: 'cd3434d87b3c46c6848a711cdf6b150c' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move scheduled
I20240501 08:14:17.907177 9014 rebalancer_tool.cc:1074] tablet f93c2d0f9c7947bb83a50c7fb8c3518a: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'a518ca5e9de24204bd028b0ce4a77d58' move scheduled
I20240501 08:14:18.484462 9014 rebalancer_tool.cc:1074] tablet a25e1ddb987b4f8c8a4031053f581c7c: '97cdee0bf9d84645a1451f9822c5d699' -> 'aba7c226aaca4e8c90931f547d48aaa3' move scheduled
I20240501 08:14:19.163479 9014 rebalancer_tool.cc:1074] tablet 5f090fda814f455a88dccd6680bdd05f: '28b7613bd4834f3ab245c82dd7e80240' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move scheduled
I20240501 08:14:19.743611 9014 rebalancer_tool.cc:1074] tablet 4db7f887af8f425c8fa661196add4d37: 'cd3434d87b3c46c6848a711cdf6b150c' -> '0a814bba1abe4653bcd418a83ec42370' move scheduled
I20240501 08:14:20.245313 9014 rebalancer_tool.cc:1074] tablet 29506d751571401f86bced7e6f5415d7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'a518ca5e9de24204bd028b0ce4a77d58' move scheduled
I20240501 08:14:20.334340 9014 rebalancer_tool.cc:1084] tablet 358b6fe9c68349cb8cb963bb3a9ed2e9: '97cdee0bf9d84645a1451f9822c5d699' -> 'aba7c226aaca4e8c90931f547d48aaa3' move ignored: Illegal state: tablet 358b6fe9c68349cb8cb963bb3a9ed2e9: replica aba7c226aaca4e8c90931f547d48aaa3 is already present
I20240501 08:14:20.334532 9014 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:14:21.392568 11111 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.25.107.62:36223, user_credentials={real_user=slave}} blocked reactor thread for 52452.3us
W20240501 08:14:22.279999 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:22.280203 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 314bb13099e14e298e69260ae58eb62c: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:22.280316 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:22.280467 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:22.280565 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:22.280776 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:22.280879 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:22.405521 9014 rebalancer_tool.cc:1074] tablet 6575b94bd51546f1bd3bb862f012e7ca: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:22.511772 9014 rebalancer_tool.cc:1074] tablet 193de3c11ae54408a64ae3007bf88920: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:22.712235 9014 rebalancer_tool.cc:1074] tablet fdaad8e1d86942ddbc97958dcf31b344: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:22.995136 9014 rebalancer_tool.cc:1084] tablet dc19d80bad46460087c143778a4b0638: 'cd3434d87b3c46c6848a711cdf6b150c' -> 'aba7c226aaca4e8c90931f547d48aaa3' move ignored: Illegal state: tablet dc19d80bad46460087c143778a4b0638: replica aba7c226aaca4e8c90931f547d48aaa3 is already present
I20240501 08:14:22.995373 9014 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:14:24.819175 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:24.819443 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:24.819586 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:24.819769 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:24.819854 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:24.819962 9014 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:25.010952 9014 rebalancer_tool.cc:1074] tablet 78d1df7b0ea44f89b9f22c156f907c1e: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:25.236610 9014 rebalancer_tool.cc:1074] tablet 45d076303ea24d7fa0fed5a9d132019c: 'aba7c226aaca4e8c90931f547d48aaa3' -> '' move scheduled
I20240501 08:14:37.714202 9014 rebalancer_tool.cc:1136] tablet 491674dfbf74498e8a787bebaa5e95da: '28b7613bd4834f3ab245c82dd7e80240' -> '0a814bba1abe4653bcd418a83ec42370' move completed: OK
W20240501 08:14:42.687908 14423 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.25.107.62:36223, user_credentials={real_user=slave}} blocked reactor thread for 61623.5us
I20240501 08:14:51.389158 9014 rebalancer_tool.cc:1136] tablet 29506d751571401f86bced7e6f5415d7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'a518ca5e9de24204bd028b0ce4a77d58' move completed: OK
I20240501 08:14:53.500077 9014 rebalancer_tool.cc:1136] tablet 4db7f887af8f425c8fa661196add4d37: 'cd3434d87b3c46c6848a711cdf6b150c' -> '0a814bba1abe4653bcd418a83ec42370' move completed: OK
I20240501 08:14:53.597628 9014 rebalancer_tool.cc:1136] tablet cfc0c7b429754d92b4dd166c073ac8ed: '97cdee0bf9d84645a1451f9822c5d699' -> 'aba7c226aaca4e8c90931f547d48aaa3' move completed: OK
I20240501 08:14:53.733417 9014 rebalancer_tool.cc:1136] tablet 6575b94bd51546f1bd3bb862f012e7ca: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
I20240501 08:14:53.830358 9014 rebalancer_tool.cc:1136] tablet 193de3c11ae54408a64ae3007bf88920: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:53.907788 9014 rebalancer_tool.cc:1136] tablet fdaad8e1d86942ddbc97958dcf31b344: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:14:54.321426 9014 rebalancer_tool.cc:1074] tablet 399d1a392b3f4d9e8531b6e0b5ad2a61: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:54.667958 9014 rebalancer_tool.cc:1074] tablet 358b6fe9c68349cb8cb963bb3a9ed2e9: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:54.795910 9014 rebalancer_tool.cc:1074] tablet e9629fbbb4554edc9ba341d42b51f252: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:54.886677 9014 rebalancer_tool.cc:1136] tablet e9629fbbb4554edc9ba341d42b51f252: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:14:54.946846 9014 rebalancer_tool.cc:1136] tablet 358b6fe9c68349cb8cb963bb3a9ed2e9: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:55.066736 9014 rebalancer_tool.cc:1136] tablet 399d1a392b3f4d9e8531b6e0b5ad2a61: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
I20240501 08:14:55.277482 9014 rebalancer_tool.cc:1136] tablet 78d1df7b0ea44f89b9f22c156f907c1e: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:14:55.351204 9014 rebalancer_tool.cc:1136] tablet 45d076303ea24d7fa0fed5a9d132019c: 'aba7c226aaca4e8c90931f547d48aaa3' -> '' move completed: OK
I20240501 08:14:55.456743 9014 rebalancer_tool.cc:1136] tablet d7903106d7e74e1ea102b1abdbfce63e: '28b7613bd4834f3ab245c82dd7e80240' -> '0a814bba1abe4653bcd418a83ec42370' move completed: OK
I20240501 08:14:57.792932 9014 rebalancer_tool.cc:1136] tablet 8b7f87b6032d45c5be5d748ab2cc71e2: '97cdee0bf9d84645a1451f9822c5d699' -> 'aba7c226aaca4e8c90931f547d48aaa3' move completed: OK
I20240501 08:14:57.897617 9014 rebalancer_tool.cc:1136] tablet a9e70954aa4e4fcfbad12e8bfe3e0059: 'cd3434d87b3c46c6848a711cdf6b150c' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move completed: OK
I20240501 08:14:58.038549 9014 rebalancer_tool.cc:1136] tablet 13b4e2d5fe4846e8b3c6081abb3c53da: 'cd3434d87b3c46c6848a711cdf6b150c' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move completed: OK
I20240501 08:14:58.126835 9014 rebalancer_tool.cc:1136] tablet a25e1ddb987b4f8c8a4031053f581c7c: '97cdee0bf9d84645a1451f9822c5d699' -> 'aba7c226aaca4e8c90931f547d48aaa3' move completed: OK
I20240501 08:14:58.267192 9014 rebalancer_tool.cc:1136] tablet 5f090fda814f455a88dccd6680bdd05f: '28b7613bd4834f3ab245c82dd7e80240' -> '19fdb76c8d1f41d7842e397adfc1ef3f' move completed: OK
I20240501 08:14:58.296332 9014 rebalancer_tool.cc:1136] tablet f93c2d0f9c7947bb83a50c7fb8c3518a: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'a518ca5e9de24204bd028b0ce4a77d58' move completed: OK
I20240501 08:14:58.426395 9014 rebalancer_tool.cc:1074] tablet 070e927b5d684367ae0b40b12a23da54: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:58.593961 9014 rebalancer_tool.cc:1074] tablet 658f2a172a2d47c8bf8b44c9d51132e4: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:58.710938 9014 rebalancer_tool.cc:1074] tablet 049090443eaa4b18ba90cb60af9a87ab: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:58.824165 9014 rebalancer_tool.cc:1074] tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:59.014489 9014 rebalancer_tool.cc:1136] tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:59.150025 9014 rebalancer_tool.cc:1136] tablet 049090443eaa4b18ba90cb60af9a87ab: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:15:02.477339 9014 rebalancer_tool.cc:1123] tablet 57ecbb30f7764fceb008baaa2c37da8c: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'a518ca5e9de24204bd028b0ce4a77d58' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:02.489373 9014 rebalancer_tool.cc:1136] tablet 658f2a172a2d47c8bf8b44c9d51132e4: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:15:02.499666 9014 rebalancer_tool.cc:1123] tablet 070e927b5d684367ae0b40b12a23da54: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
Illegal state: tablet server 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): unacceptable health status UNAVAILABLE
W20240501 08:15:04.110802 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:04.175048 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:04.186753 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:04.235409 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:04.259295 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:15:04.339462 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:04.353495 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:04.363929 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:04.364804 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:04.365110 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:04.415828 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:04.416220 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:04.458058 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:15:04.464892 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:15:04.465955 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:04.486106 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 08:15:04.487216 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:15:04.503033 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:04.544868 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:04.549856 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:04.556521 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:04.567695 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240501 08:15:04.571534 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:04.572645 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:04.578285 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:04.599027 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:04.632755 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:15:04.697484 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:04.700100 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:04.710034 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:04.711031 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:15:04.743072 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:15:04.792706 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:04.796468 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:04.805277 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:04.838220 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:04.839347 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:04.867714 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:04.890120 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:04.910184 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:04.942667 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:04.945299 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:04.966581 17927 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:04.966781 17927 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:04.967535 17927 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:04.982474 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:04.993283 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:05.020476 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:05.022555 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:05.055418 8895 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 59962.9us
W20240501 08:15:05.056535 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:05.056787 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:05.057152 17927 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:15:05.063946 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:15:05.064054 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:05.073711 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:05.081180 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:05.127185 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:05.128079 17877 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:05.128206 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:05.128264 17877 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:05.128938 17877 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:05.136727 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:05.143648 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:05.143838 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:05.148165 17877 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:15:05.219732 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:15:05.360143 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:05.362825 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:05.363898 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:05.405043 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:05.409350 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:05.410056 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:05.426535 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:05.468364 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:15:05.637934 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:05.652191 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:05.680420 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:05.696483 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:05.700850 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:05.740695 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:05.789156 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:05.817144 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:05.818360 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:05.860008 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:05.866748 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:05.866904 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:05.988842 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:06.114061 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:06.116591 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:06.116640 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:06.119693 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:06.133399 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:06.152066 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:06.179148 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:06.293751 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:06.311465 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:06.323944 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:06.325760 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:06.356319 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:06.358736 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:06.389326 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:06.411873 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:06.418056 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:06.445402 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:06.546411 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:06.551524 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:06.552699 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:06.570456 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:06.579766 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:06.615315 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:06.661754 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:06.747946 17927 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:06.748157 17927 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:06.748894 17927 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:06.753163 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:15:06.766219 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:06.766410 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:06.766703 17927 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:15:06.866393 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:15:06.870599 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:06.881839 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:06.899080 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:06.906412 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:15:06.908792 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:06.912700 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:06.924293 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:06.924958 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:06.936673 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:06.956243 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:06.975898 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:06.976884 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:06.996001 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:06.997017 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:07.010413 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:07.026885 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:07.079105 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:07.166306 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:07.171866 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:07.181128 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:07.206137 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:07.239197 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:07.257086 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:07.260931 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:07.287530 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:07.291286 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:15:07.308493 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:07.343184 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:07.358683 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:07.413247 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:07.424854 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:07.456238 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:07.463122 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:15:07.467882 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:07.546919 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:15:07.547724 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:07.566076 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:07.572239 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:15:07.603546 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:07.639698 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:07.640039 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:15:07.661823 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:07.710731 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:07.716836 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:07.719543 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:07.724184 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:07.758565 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:07.773892 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:07.781397 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:07.791602 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:07.838836 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:07.839131 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:15:08.020272 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:08.027180 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:08.033068 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:08.052398 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:08.064466 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:08.069556 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:08.129899 18227 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:08.130095 18227 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:08.130847 18227 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:08.135867 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:15:08.136515 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:08.136624 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:08.136854 18227 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:15:08.300920 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:08.322453 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:08.323697 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:08.347725 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:08.382426 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:08.392835 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:08.406011 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:08.410539 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:08.427376 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:08.451316 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:08.483665 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:08.505964 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:08.508922 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:08.702389 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:08.709328 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:08.722537 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:08.733978 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:08.735690 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:08.737964 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:08.741501 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:08.764165 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:08.773245 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:08.833582 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:08.929857 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:08.971155 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:09.064399 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:09.088341 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:09.099025 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:09.140944 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:09.155948 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:09.193130 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:09.232883 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:09.240381 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:09.251439 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:09.276432 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:09.280946 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:09.321084 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:09.330605 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:09.347568 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:09.361830 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:09.422863 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:09.507133 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:15:09.517014 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:09.522222 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240501 08:15:09.523836 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:09.524937 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:09.536146 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:09.551204 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:09.559689 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:15:09.568089 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:09.572440 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:09.575671 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:09.612310 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:09.645162 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:09.673163 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:09.692425 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:15:09.748670 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:09.755205 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:09.779565 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:09.802841 18736 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:09.803057 18736 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:09.803741 18736 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:09.820267 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:09.847678 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240501 08:15:09.850831 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:09.852384 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:09.852593 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:09.852908 18736 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:15:09.870741 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:09.915877 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:09.925431 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:09.938149 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:09.989533 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:10.010075 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:10.013520 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:10.034576 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:10.059276 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:10.065225 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:10.073110 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:10.094291 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:10.108021 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:10.116638 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:10.121805 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:10.124393 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:10.125094 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:10.147801 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:10.150476 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:10.177562 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:15:10.306370 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:15:10.387486 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:10.391150 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:10.393401 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:10.403054 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:10.415513 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:10.428341 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:10.443802 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:10.689425 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:10.714398 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:10.731343 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:10.743058 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:10.749493 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:10.767859 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:10.778088 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:10.792925 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:10.829633 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:10.833214 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:10.896504 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:10.904565 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:10.930433 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:11.126988 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:11.127106 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:11.132308 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:11.134454 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:11.148458 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:11.149387 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:11.178969 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:11.257627 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:11.366003 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:11.371295 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:11.372799 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:11.388510 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:11.397063 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:11.410858 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:11.422760 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:11.553457 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:11.563428 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:11.569167 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:11.569928 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:11.577472 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:11.583832 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:11.588780 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:11.588892 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:11.670279 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:11.675366 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:11.741111 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:11.862653 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:11.868482 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:11.879078 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:11.879462 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:11.889142 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:11.891995 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:11.893848 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:11.932386 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:11.953254 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:12.078428 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:12.081940 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:12.092231 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:12.105041 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:12.112179 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:12.124418 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:12.125406 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:12.125963 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:12.138486 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:12.143221 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:12.148658 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:12.212496 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:12.233906 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:12.234380 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:12.268087 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:12.324668 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:12.352644 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:12.359192 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:12.362942 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:12.363309 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:12.374536 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:12.377405 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:12.393103 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:12.483628 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:12.495640 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:15:12.609019 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:12.651456 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:12.688896 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:12.693969 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:12.700938 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:12.718575 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:12.721246 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:12.722582 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:12.748375 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:12.761268 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:12.766634 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:12.791136 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:12.849524 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:12.855253 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:15:12.874012 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:12.882583 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:12.957051 18729 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:12.957238 18729 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:12.957903 18729 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:12.959604 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:12.959816 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:12.959919 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:12.960156 18729 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:15:13.035897 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:13.069674 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:13.071843 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:13.107467 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:13.107564 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:13.146399 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:13.158691 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:13.273399 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:13.378461 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:13.398849 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:13.399237 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:13.414880 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:13.457087 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:13.457957 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:13.560829 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:13.571085 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:13.572419 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:13.595353 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:13.595345 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:13.604249 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:13.686992 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:13.772140 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:13.825490 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:13.870030 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:13.881121 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:13.881291 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:13.898010 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:13.915083 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:13.922577 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:13.942974 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:13.953020 19109 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:13.953212 19109 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:13.953886 19109 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:14.009577 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:14.057252 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:15:14.057456 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:14.057579 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:14.058037 19109 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:15:14.079736 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:14.091358 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:14.103926 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:14.106065 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:14.106087 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:14.135743 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:14.163484 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:14.166039 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:14.198336 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:14.223294 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:14.306149 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:14.309458 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:14.327533 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:14.328892 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:14.346426 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:14.350944 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:14.362541 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:14.432027 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:14.447043 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:14.480326 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:14.567198 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:14.584510 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:14.609185 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:15:14.627682 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:14.632989 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:14.651391 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:14.657994 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:14.663192 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:14.669941 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:14.683513 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:14.699379 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:14.702847 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 08:15:14.723127 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:14.730669 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:15:14.732316 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:14.739558 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:14.740818 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240501 08:15:14.765348 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:14.770911 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:14.787113 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:14.797364 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:14.810056 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:14.826648 19447 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:14.826864 19447 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:14.827534 19447 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:14.831323 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:15:14.831609 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:14.831701 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:14.831919 19447 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:15:14.866108 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:14.933681 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:14.998457 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:15.003175 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:15.013433 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:15.023195 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:15.025491 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:15.034905 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:15.056447 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:15.068960 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:15.098980 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:15.128371 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:15.156477 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:15.233381 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:15:15.284701 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:15:15.299250 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:15.312527 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:15.323876 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:15.350994 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:15.356357 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:15.381127 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:15.383862 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:15.399566 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:15.421909 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:15.458015 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:15.474961 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:15.483976 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:15.508741 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:15.533188 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:15.691438 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:15.694469 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:15.699115 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:15.708701 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:15.720369 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:15.728557 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:15.912448 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:15.928319 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:15.928704 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:15.929114 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:15.931036 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:15.931546 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:15.957615 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:16.145876 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:15:16.163110 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:16.172353 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:16.180827 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:16.183712 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:16.196148 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:16.202117 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:16.357708 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:16.363008 19829 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:16.363219 19829 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:16.363932 19829 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:16.367062 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:16.367722 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:16.367825 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:16.368062 19829 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:15:16.413880 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:16.413879 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:16.441293 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:16.453910 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:16.460156 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:16.474895 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:16.479496 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:16.480904 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:16.488139 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:16.489779 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:16.522738 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:15:16.524561 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:16.550086 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:16.551041 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:16.595566 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:16.627880 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:16.644253 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:16.793871 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:16.796736 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:16.806440 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:16.811156 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:16.825045 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:16.827304 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:16.829119 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:16.858448 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:16.941193 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:17.057925 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:17.059381 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:15:17.114171 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:17.118342 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:17.119264 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:17.147938 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:17.149091 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:17.154354 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:17.172073 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:17.184725 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:17.243330 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:17.246717 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:17.262467 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:17.268389 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:17.269349 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:17.277335 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:15:17.277591 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:17.337217 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:17.337579 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:17.341321 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:17.345185 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:17.356729 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:17.356976 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:17.403524 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:17.530894 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:17.530978 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:17.550694 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:17.573632 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:17.613010 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:15:17.630054 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:17.636256 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:17.648893 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:15:17.654932 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:17.661036 6833 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 105313us
W20240501 08:15:17.734923 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:17.779394 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:17.850081 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:17.859706 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:17.885751 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:17.888433 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:17.920104 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:17.926276 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:15:17.928298 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:17.945273 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:18.132288 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:18.136519 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:18.137112 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:18.155964 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:18.172158 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:18.175097 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:18.237962 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:18.351811 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:18.357658 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:18.375365 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:18.429081 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:18.430727 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:18.452798 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:18.548918 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:18.550240 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:18.560546 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:18.568032 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:18.618680 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:18.624873 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:15:18.649461 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:18.749609 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:18.862017 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:18.863499 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:18.866521 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:18.881757 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:18.884685 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:18.887360 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:18.992420 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:19.057797 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:19.079991 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:19.112957 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:19.127334 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:15:19.155949 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:19.171552 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:19.184494 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:19.205263 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:19.205693 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:19.252058 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:19.253198 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:19.283226 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:19.298136 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:15:19.310417 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:19.331462 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:19.338392 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:19.363423 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:19.383193 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:19.399253 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:19.404304 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:15:19.566161 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:19.568962 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:19.603147 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:19.609047 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:19.630971 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:19.650563 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:19.700397 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:15:19.723551 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:15:19.726975 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:19.751049 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 08:15:19.753952 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:19.770118 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:19.796617 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:15:19.797797 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:19.823549 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:19.851528 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:19.862756 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:19.866817 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:15:19.873772 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:19.885066 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:19.900576 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:19.925271 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:19.933854 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:19.956763 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:19.992976 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:15:19.994212 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:20.017573 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:20.064126 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:20.075466 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:20.086302 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:15:20.117467 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:20.125696 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:20.140872 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:20.143050 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:20.165738 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:20.184376 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:20.184836 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:20.187569 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:15:20.284020 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:20.322873 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:15:20.372843 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:20.375530 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:20.394390 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:20.423923 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:20.438655 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:20.492800 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:20.629035 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:20.632202 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:20.677111 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:20.699064 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:20.711798 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:20.717973 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:20.718668 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:20.829039 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:20.858809 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:20.876583 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:20.880944 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:20.894752 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:20.898715 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:20.983008 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:15:21.130575 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:21.146894 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:21.159288 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:21.161320 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:21.174510 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:21.175817 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:21.202872 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:21.303506 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:21.314388 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:21.320694 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:21.328066 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:21.328964 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:21.353411 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:21.456748 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:21.537603 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:21.586532 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:15:21.626935 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:21.635104 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:21.637755 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:15:21.644043 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:21.644737 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:15:21.645119 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:21.658990 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:21.700455 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:21.762996 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:21.788170 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:21.812405 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:15:21.924635 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:21.942363 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:21.945942 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:21.994987 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:21.995874 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:22.010524 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:22.011122 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:22.027094 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:22.081512 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:22.152812 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:22.153584 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:22.156286 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:22.162024 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:22.167969 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:22.168429 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:22.221386 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:22.241793 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:22.273555 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:22.329785 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:22.337720 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:15:22.387749 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:22.387921 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:15:22.399284 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:22.426595 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:22.441547 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:22.453950 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:22.461983 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:15:22.463264 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:22.508605 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:22.540614 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:22.556651 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:15:22.584440 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:15:22.588275 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:22.590185 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:15:22.604552 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:22.612861 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:22.623178 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:15:22.644397 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:22.645283 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:22.651643 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:22.667433 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:22.686684 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:22.694861 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:15:22.853683 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:22.863178 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:22.864014 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:22.873591 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:22.900394 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:22.901870 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:22.935935 20551 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:22.936136 20551 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:22.936858 20551 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:22.940924 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 08:15:22.942145 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:22.947580 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:22.947723 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:22.947980 20551 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:15:22.982460 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:23.168530 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:23.191411 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:23.209576 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:23.213665 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:23.227044 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:23.232689 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:23.233724 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:23.236337 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:23.243192 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:23.283902 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:23.306838 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:23.341269 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:23.385798 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:23.547502 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:23.552737 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:23.561468 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:23.577412 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:23.577571 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:23.589308 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:23.598738 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:23.748576 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:23.849784 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:23.868228 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:23.878407 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:23.897053 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:23.904560 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:23.908581 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:23.924669 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:23.931491 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:23.932591 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:23.961733 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:24.001322 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:24.020697 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:24.053280 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:24.056411 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:24.067167 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:24.081316 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:15:24.209128 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:24.214735 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:24.343784 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:15:24.396865 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:15:24.433032 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:24.484061 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:24.491528 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:24.522614 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:24.524118 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:24.524765 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:24.539616 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:15:24.557713 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:24.591109 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:24.646824 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:24.659622 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:24.662432 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:24.683739 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:15:24.687074 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:24.718355 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:24.735412 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:24.739660 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:24.780138 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 08:15:24.781085 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:24.789644 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:15:24.828456 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:15:24.835603 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:24.860054 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:24.885700 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:15:24.902410 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:24.906404 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:24.915591 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:15:24.919173 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:15:24.922358 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:24.924422 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:24.935843 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:24.940358 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:24.949290 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:24.965613 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:25.068970 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:25.131870 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:15:25.156592 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:25.170722 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:25.194466 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:25.195405 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:25.199683 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:25.204114 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:25.245498 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:25.259656 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:15:25.347522 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:15:25.362617 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:25.374370 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:25.381506 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:25.384400 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:25.399474 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:25.405503 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:25.453012 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:25.570406 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:25.658908 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:25.681083 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:25.724627 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:25.735548 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:25.740061 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:25.751279 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:25.762001 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:25.813472 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:25.822731 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:25.840377 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:25.851954 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:25.853364 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:25.876184 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:26.007274 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:26.075978 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:15:26.079466 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:26.092278 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:26.100950 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:26.101634 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:26.110930 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:26.128410 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:26.297775 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:15:26.337177 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:26.371340 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:26.389066 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:26.390390 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:26.410090 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:26.414371 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:26.419586 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:15:26.425484 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:26.468627 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:26.543066 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:26.545629 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:15:26.546116 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:26.546883 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:26.549806 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:26.557600 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:26.558537 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:26.592922 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:26.769868 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:26.770916 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:26.776638 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:26.777778 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:26.798518 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:26.811308 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:26.832638 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:26.842608 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:26.996526 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:27.003247 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:27.021522 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:27.027766 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:27.035511 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:27.054556 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:27.069864 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:15:27.094501 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:27.143504 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:27.231415 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:27.246941 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:27.247574 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:27.249987 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:27.254899 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:27.258900 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:27.267534 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:27.268191 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:27.284050 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:27.300717 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:15:27.316079 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:27.337721 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:27.446573 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:15:27.506942 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:27.540500 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:27.541378 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:27.558338 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:27.566136 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:27.575634 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:27.582757 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:27.613081 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:27.613494 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:27.630702 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:27.637683 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:15:27.642048 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:27.658197 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:27.690290 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:27.709975 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:27.716989 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:27.753165 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:15:27.779772 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:27.832587 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:15:27.844687 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:15:27.927362 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:27.951462 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:27.952850 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:27.955260 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:27.957441 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:28.002322 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:28.222620 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:28.230906 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:15:28.253582 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:28.264747 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:28.272099 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:28.281778 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:28.304877 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:28.316790 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:28.334244 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:28.346776 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:28.348860 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:28.365368 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:28.382891 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:28.388968 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:28.464133 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:28.564880 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:28.627795 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:28.644163 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:28.645881 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:28.661949 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:28.669607 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:28.719072 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:28.720626 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:28.822093 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:28.928241 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:28.937253 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:28.954284 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:28.984949 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:28.988333 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:29.013026 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:29.018154 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:29.032661 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:29.074920 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:29.102527 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:29.106820 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:29.113443 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:29.113624 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:15:29.114945 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:29.120654 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:29.120970 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:29.209388 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:29.303421 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:29.310528 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:29.327891 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:29.335165 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:29.353942 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:29.366498 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:29.488675 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:29.547966 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:29.548298 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:29.557381 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:29.559046 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:29.574704 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:29.599895 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:29.610648 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:29.664752 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:29.707823 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:29.726047 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:29.787840 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:15:29.788887 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:29.795526 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:29.804869 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:29.804927 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:29.809454 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:29.823031 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:29.828316 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:29.845595 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:15:29.846674 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:29.906240 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:29.914695 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:29.924085 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:15:29.925110 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:29.934778 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:29.989133 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:15:30.015931 21926 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:30.016127 21926 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:30.016781 21926 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:30.019049 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:15:30.019305 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:30.019408 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:30.019676 21926 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:15:30.026512 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:30.072552 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:15:30.105187 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:30.106693 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:30.127024 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:30.142066 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:30.145161 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:30.146957 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:30.166867 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:30.171561 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:30.224054 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:30.236028 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:30.268783 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:30.286790 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:30.302335 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:30.333262 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:30.365819 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:30.461695 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:15:30.491273 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:30.506505 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:30.507470 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:30.508072 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:30.517045 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:30.518760 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:30.540982 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:30.582943 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:15:30.590401 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:30.762606 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:30.786293 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:30.791524 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:30.795058 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:30.810057 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:30.825663 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:30.835909 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:30.939114 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:30.975653 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:30.988763 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:30.998615 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:31.003258 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:31.004729 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:31.031127 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:31.049633 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:15:31.159560 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:31.203878 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:31.207444 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:31.210754 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:31.226430 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:31.234561 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:31.250126 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:31.363310 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:31.494859 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:31.509109 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:31.519512 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:31.538064 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:31.554801 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:31.555397 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:31.569800 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:31.584657 6616 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:31.593726 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:31.600723 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:31.601042 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:31.609318 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:31.609928 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:31.660104 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:31.689155 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:31.691082 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:31.698135 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:31.890233 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:31.894825 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:31.899549 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:31.906419 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:31.907205 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:31.910779 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:32.099278 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:32.128935 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:32.135783 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:32.140551 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:32.145965 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:32.146173 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:15:32.146304 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:32.147856 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:32.170861 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:32.178895 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:32.193682 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:32.203289 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:32.278347 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:32.359719 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:32.364183 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:32.377939 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:32.391849 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:32.394697 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:32.395561 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:32.399957 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:15:32.444490 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:32.457532 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:32.501914 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:32.530445 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:15:32.620810 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:15:32.638342 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:32.658921 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:32.690220 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:32.691020 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:32.694509 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:32.718845 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:32.718845 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:32.746007 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:32.759776 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:32.768767 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:32.782986 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:32.791779 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:32.801122 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:32.829998 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 82410.6us
I20240501 08:15:32.836014 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:32.848047 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:32.868784 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:15:33.055121 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:15:33.067019 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:33.074694 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:33.079242 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:33.094938 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:33.097923 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:33.129745 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:33.142495 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:33.348611 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:33.360522 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:33.365221 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:33.376888 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:33.384560 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:33.400740 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:33.414559 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:33.418144 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:33.433844 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:33.441027 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:33.467653 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:33.487465 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:33.518472 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:33.523861 20551 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:33.524042 20551 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:33.524725 20551 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:33.526382 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240501 08:15:33.529604 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:33.542793 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:15:33.565539 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:33.565775 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:33.566175 20551 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:15:33.719981 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:33.722694 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:33.745048 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:33.749790 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:33.756419 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:33.756660 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:33.761140 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:33.785004 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:33.917984 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:33.952742 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:33.969882 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:34.001647 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:34.047348 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:34.063963 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:34.100131 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:34.120213 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:34.151077 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:34.168875 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:34.172814 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:34.184046 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:34.194646 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:34.202593 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:34.214828 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:34.253381 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:15:34.279826 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:34.334030 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:34.370741 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:34.401299 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:34.402508 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:34.413540 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:34.418100 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:34.426931 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:34.442565 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:34.554596 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:34.555239 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:34.571573 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:34.673329 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:34.688858 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:34.694489 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:34.697278 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:34.711206 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:34.720453 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:34.740413 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:34.802012 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 08:15:34.808140 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:34.848330 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:34.876538 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:15:34.876796 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:34.881140 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:15:34.903606 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:34.907815 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:34.909921 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:34.926359 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:34.926954 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:34.928735 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:34.929381 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:34.943476 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:34.944121 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:34.970232 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:15:35.011796 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:15:35.068217 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:15:35.069133 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:15:35.121100 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:35.192786 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:35.204473 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:35.206689 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:35.221596 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:35.223196 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:35.242467 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:35.245955 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:35.249930 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:35.266141 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:35.281165 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:35.287055 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:35.308270 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:35.317178 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:35.338436 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:35.346005 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:35.549573 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:35.560222 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:35.566464 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:35.579197 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:35.586812 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:35.590602 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:35.627657 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:35.639413 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:35.863394 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:35.870585 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:35.891467 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:35.902621 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:15:35.907321 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:35.930277 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:35.952817 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:36.008513 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:36.011766 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:36.040870 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:36.062556 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:36.071988 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:36.113399 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:36.117385 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:36.131356 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:36.295686 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:36.301005 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:36.313385 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:36.313397 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:36.316797 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:36.317500 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:36.319602 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:36.370846 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:36.423856 22887 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:36.424064 22887 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:36.424760 22887 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:36.427028 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:36.430214 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:36.430351 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:36.430876 22887 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:15:36.476420 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:36.533571 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:36.541043 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:36.545897 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:36.556865 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:36.596351 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:36.607642 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:36.608764 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:36.624552 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:36.651382 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:36.778978 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:36.779925 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:36.791153 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:36.793138 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:36.794524 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:36.798485 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:36.799525 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:36.820813 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:36.840896 6833 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 43793.8us
W20240501 08:15:36.883569 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:15:37.007493 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:37.024160 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:37.077317 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:37.095062 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:37.122306 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:37.125599 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:37.153013 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:37.168908 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:37.207459 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:37.215021 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:37.219190 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:37.259164 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:37.266021 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:37.292492 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:37.309793 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:37.323348 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:37.323675 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:15:37.353780 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:37.353964 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:37.397298 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:37.456027 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:37.463276 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:37.506121 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:37.519155 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:37.578662 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:37.580385 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:37.582530 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:37.606146 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:37.628280 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:15:37.631230 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:37.635386 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:37.636693 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:37.702145 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:15:37.885962 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:37.890012 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:37.893188 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:37.898339 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:37.899179 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:37.924528 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:37.926201 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:37.961354 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:38.001693 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:38.004527 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:38.039988 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:38.046401 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:38.047142 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:38.074090 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:38.088025 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:38.263975 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:38.279623 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:38.292856 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:38.303416 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:38.320926 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:38.336038 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:38.343284 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:15:38.565624 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:38.566161 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:38.575475 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:38.581513 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:38.598222 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:38.611399 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:38.656561 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:38.657496 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:38.670800 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:38.679281 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:38.695163 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:38.716799 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:38.763273 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:38.786341 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:38.794643 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:38.877792 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:15:38.970156 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:15:39.018213 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:39.022661 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:15:39.032724 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:39.039011 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:39.044696 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:39.050312 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:39.051869 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:39.053160 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:39.241171 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:39.264338 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:39.273909 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:39.283262 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:39.311385 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:15:39.316130 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:39.332676 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:39.336843 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:39.370460 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:15:39.450359 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:39.456233 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:39.460729 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:39.477546 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:39.497180 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:39.500160 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:15:39.515084 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:39.523787 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:39.531206 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:39.557463 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:39.741583 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:39.747594 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:39.748447 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:15:39.769423 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:15:39.771103 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:39.773891 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:39.781702 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:39.792253 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:39.834510 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:39.858668 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:15:39.863641 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:15:39.882447 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:15:39.884976 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:15:39.892000 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:39.920336 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:39.921229 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:39.933851 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:39.941366 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:39.942181 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:39.972028 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:39.973384 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:40.012629 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:40.049208 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:15:40.066929 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:15:40.181284 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:15:40.197497 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:40.198590 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240501 08:15:40.224615 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:40.227831 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:40.233364 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:40.248042 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:40.248652 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:40.265280 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:40.412026 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:40.452420 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:40.486315 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:40.518447 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:40.521339 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:40.591409 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:40.601585 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:40.604794 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:40.616477 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:40.621299 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:40.623613 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:40.643016 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:40.653069 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:40.690622 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:40.698619 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:40.814199 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:40.832294 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:40.852908 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:15:40.920397 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:40.937979 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:40.944490 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:40.950433 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:40.956581 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:40.967943 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:41.071167 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:41.167275 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:15:41.227702 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:41.236344 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:41.246111 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:41.269254 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:41.286898 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:41.287356 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:41.317755 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:41.318440 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:41.336844 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:41.352128 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:41.375051 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:41.396064 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:41.412336 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:41.488466 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:41.501461 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:41.505374 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:41.690101 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:41.694492 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:41.704183 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:41.705096 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:41.706171 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:41.706702 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:41.713893 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:41.724448 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:41.781921 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:41.814827 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:15:41.888713 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:41.963797 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:41.997532 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:42.002039 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:42.014724 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:42.021376 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:42.021657 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:42.053503 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:42.118218 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:42.128978 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:15:42.154608 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:42.168987 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:42.179100 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:42.182845 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:42.188375 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:42.200155 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:42.219825 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:42.241791 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:42.267604 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:42.374697 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:42.375547 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:15:42.380225 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:42.384374 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:42.388955 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:42.394613 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:42.406693 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:42.410451 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:42.443948 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:15:42.512235 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:42.600729 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:42.602366 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:42.626597 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:42.641258 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:42.663142 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:42.668771 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:42.676115 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:42.680449 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:42.710327 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:42.757863 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:42.846225 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:42.857558 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:42.863903 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:42.866691 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:42.881700 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:42.900373 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:43.058032 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:43.138999 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:43.142443 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:43.142903 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:43.214447 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:43.215960 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:43.242893 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:43.271816 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:43.271940 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:15:43.333479 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:43.343410 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:43.345968 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:43.352633 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:43.373103 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:43.387030 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:43.401346 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:15:43.406275 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:43.475271 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:43.616515 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:43.625334 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:43.643592 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:43.671044 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:43.689783 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:43.697800 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:43.722095 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:43.823798 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:43.898308 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:15:43.902108 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:43.904899 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:43.906191 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:43.949146 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:43.954138 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:44.000968 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:44.099238 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:44.100222 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:15:44.167053 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:15:44.168124 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:44.168466 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:44.170372 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:44.172480 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:44.172993 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:44.176214 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:44.216224 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:44.310060 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:15:44.410100 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:44.411653 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:15:44.418886 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:44.433717 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:44.438900 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:44.456002 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:44.461853 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:44.478883 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:44.481396 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:44.527484 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:44.583797 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:44.607750 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:44.610683 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:44.632143 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:44.639787 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:44.649367 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:44.706099 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:15:44.757333 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:44.775200 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:15:44.870927 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:15:44.873994 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:44.874501 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:44.877732 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:44.887730 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:44.889187 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:44.903352 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:44.906620 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:44.914736 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:15:44.917147 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:44.926347 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:44.945158 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:44.987921 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:45.132285 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:15:45.160921 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:45.168907 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:45.169788 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:45.193550 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:45.193899 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:45.201649 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:45.216933 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:45.244272 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:15:45.301977 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:45.378094 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:45.378420 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:45.384943 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:45.386543 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:45.392340 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:45.407320 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:45.430163 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:45.448624 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:45.539541 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:15:45.625141 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:15:45.646911 6486 maintenance_manager.cc:419] P d978dc7272e947d9938cb39d0d6c1eb8: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033394
I20240501 08:15:45.668988 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:45.670866 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:45.697238 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:45.723219 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:45.731632 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:45.739128 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:45.770716 6485 maintenance_manager.cc:643] P d978dc7272e947d9938cb39d0d6c1eb8: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.123s user 0.037s sys 0.007s Metrics: {"bytes_written":20240,"cfile_init":1,"dirs.queue_time_us":8016,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":3744,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1048,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":42441,"threads_started":2,"wal-append.queue_time_us":31555}
I20240501 08:15:45.791596 6486 maintenance_manager.cc:419] P d978dc7272e947d9938cb39d0d6c1eb8: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7467 bytes on disk
I20240501 08:15:45.794371 6485 maintenance_manager.cc:643] P d978dc7272e947d9938cb39d0d6c1eb8: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
W20240501 08:15:45.810357 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:45.919314 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:15:45.953374 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:45.962134 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:45.966693 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:45.994518 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:46.002490 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:46.003322 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:46.043099 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:46.046803 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:46.281044 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:46.307241 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:46.325160 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:46.333024 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:46.342357 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:46.360550 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:46.361027 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:15:46.387670 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:46.404613 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:46.489993 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:46.490588 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:46.495764 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:46.499878 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:46.541431 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:46.547837 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:46.577101 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 49753.6us
W20240501 08:15:46.632225 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:46.738785 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:46.785475 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:46.816358 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:46.829478 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:46.854971 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:46.869989 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:46.877591 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:46.881124 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:46.916760 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:46.916831 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:46.917398 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:46.933612 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:46.964720 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:47.054633 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:47.097846 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:47.099298 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:47.119311 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:47.187844 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:47.193651 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:47.211479 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:47.229789 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:15:47.235203 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:47.235872 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:47.258316 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:47.343451 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:47.376920 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:47.384135 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:47.404836 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:47.414559 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:47.419734 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:47.426204 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:47.443765 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:47.584546 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:47.600994 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:15:47.679150 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:47.680083 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:47.690687 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:47.705363 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:47.709532 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:47.726397 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:47.770228 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:47.834488 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:47.850798 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:47.851802 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:47.873612 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:47.883600 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:47.906921 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:47.934301 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:47.956943 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:47.991264 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:48.074874 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:15:48.098811 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:15:48.100400 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:48.102950 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:48.119167 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:48.132069 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:48.138051 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:48.160475 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:48.285831 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:48.352437 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:48.360879 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:48.370524 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:48.377276 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:48.390413 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:48.425499 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:48.438975 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:48.446694 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:48.453787 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:48.552882 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:48.561473 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:48.566500 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:48.594822 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:48.600497 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:48.605960 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:48.902626 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:48.923647 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:48.926857 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:48.940361 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:48.947691 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:48.956023 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:48.980053 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:49.025600 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:49.026361 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:49.049685 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:49.090698 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 48997.8us
W20240501 08:15:49.132637 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:49.133760 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:49.168797 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:49.217392 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:49.247972 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:49.259785 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:49.266423 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:49.267069 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:49.270715 6702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.3:43505, user_credentials={real_user=slave}} blocked reactor thread for 49378.4us
I20240501 08:15:49.277212 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:49.278453 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:49.299008 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:49.327641 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:49.522146 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:49.522747 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:49.528379 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:49.537189 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:49.551712 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:49.571353 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:49.579326 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:49.592339 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:49.596714 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:49.619552 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:49.619611 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:49.628981 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:49.631222 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:49.637470 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:49.679407 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:49.686936 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:49.704440 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:49.722748 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:49.903352 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:49.910534 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:15:49.922286 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:49.922983 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:49.923372 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:49.926422 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:49.931046 25172 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:49.931227 25172 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:49.931882 25172 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:49.933437 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:49.942792 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:49.954092 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:15:49.954373 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:49.954667 25172 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:15:49.961726 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:50.015717 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:50.040601 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:15:50.042109 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:50.184216 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:15:50.210134 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:50.234354 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:50.235791 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:50.242100 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:50.269475 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:50.273123 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:50.290299 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:50.295339 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:50.311617 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:50.321506 6704 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 69889.9us
W20240501 08:15:50.327122 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:15:50.328418 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:50.340674 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:50.342592 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:50.347298 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:50.402372 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:50.405324 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:15:50.424189 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:50.561646 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:50.596477 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:50.614331 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:50.616429 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:50.632717 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:50.646528 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:50.651885 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:50.652088 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:50.657759 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:50.897578 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:15:50.913166 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:50.922279 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:50.940055 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:50.992058 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:50.992723 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:50.998028 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:50.999399 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:51.004451 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:51.005909 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:51.019510 25082 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:51.019704 25082 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:51.025513 25082 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:15:51.044415 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:51.067965 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240501 08:15:51.075140 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:51.074190 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:15:51.079953 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:51.080185 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:51.080605 25082 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:15:51.085475 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:51.111617 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:51.124277 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:51.143390 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:51.331408 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:51.343727 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:51.348814 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:51.354519 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:51.361071 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:51.372465 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:51.373230 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:51.390362 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:51.583415 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:51.595508 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:51.596510 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:51.613354 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:51.617906 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:51.618824 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:51.619694 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:51.639477 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:51.659806 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:51.721505 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:51.778898 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:51.789305 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:51.815403 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:51.823036 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:51.840881 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:51.849190 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:51.850088 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:51.853485 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:51.884143 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:51.914379 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:52.007190 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:52.098264 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:52.145891 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:52.168964 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:52.175372 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:52.183593 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:52.206151 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:52.227552 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:52.228232 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:52.232566 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:52.244165 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:52.246564 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:52.264132 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:52.264484 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:52.301975 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:52.307809 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:52.381389 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:52.414019 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:52.502291 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:52.556020 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:52.576066 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:52.578742 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:52.593015 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:52.595645 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:52.611153 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:52.635732 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:52.645879 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:52.656088 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:52.687553 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:52.788591 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:52.876895 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:52.898418 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:52.927642 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:52.989320 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:52.992902 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:53.077459 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:53.084084 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:15:53.096104 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:53.158933 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:53.170729 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:53.181046 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:15:53.183617 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:53.187022 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:53.187929 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:53.224660 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:53.225296 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:53.441713 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:53.455873 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:53.467788 6703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 49549.5us
W20240501 08:15:53.485286 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:53.487545 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:53.493211 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:53.506052 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:53.522961 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:53.527508 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:53.559990 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:53.644480 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:53.674419 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:53.700345 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:53.700953 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:53.717831 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:53.726632 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:53.777834 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:53.783387 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:53.910198 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:53.926012 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:53.927881 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:53.933179 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:53.951409 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:53.984000 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:53.985186 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:54.076844 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling FlushMRSOp(8b7f87b6032d45c5be5d748ab2cc71e2): perf score=0.033353
I20240501 08:15:54.084180 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(5f090fda814f455a88dccd6680bdd05f): perf score=0.033357
W20240501 08:15:54.094889 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:54.113615 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:15:54.145502 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:15:54.231974 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:54.256263 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:54.266969 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:54.294395 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: FlushMRSOp(8b7f87b6032d45c5be5d748ab2cc71e2) complete. Timing: real 0.217s user 0.065s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":337,"dirs.queue_time_us":1952,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":933,"drs_written":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4,"lbm_write_time_us":846,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":27614,"threads_started":3,"wal-append.queue_time_us":13368}
I20240501 08:15:54.300773 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling FlushMRSOp(070e927b5d684367ae0b40b12a23da54): perf score=0.033409
W20240501 08:15:54.308967 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:54.310503 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(5f090fda814f455a88dccd6680bdd05f) complete. Timing: real 0.226s user 0.062s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":505,"dirs.queue_time_us":5812,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1045,"drs_written":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4,"lbm_write_time_us":940,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":23331,"threads_started":3,"wal-append.queue_time_us":1945}
I20240501 08:15:54.312765 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(5f090fda814f455a88dccd6680bdd05f): 4834 bytes on disk
I20240501 08:15:54.318424 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(5f090fda814f455a88dccd6680bdd05f) complete. Timing: real 0.004s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240501 08:15:54.369650 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:54.408624 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: FlushMRSOp(070e927b5d684367ae0b40b12a23da54) complete. Timing: real 0.107s user 0.062s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":326,"dirs.run_cpu_time_us":263,"dirs.run_wall_time_us":2338,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":935,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7671,"threads_started":1,"wal-append.queue_time_us":7796}
I20240501 08:15:54.410141 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(8b7f87b6032d45c5be5d748ab2cc71e2): 4921 bytes on disk
I20240501 08:15:54.411497 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(8b7f87b6032d45c5be5d748ab2cc71e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":144,"lbm_reads_lt_1ms":4}
I20240501 08:15:54.412883 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(070e927b5d684367ae0b40b12a23da54): 4581 bytes on disk
I20240501 08:15:54.414233 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(070e927b5d684367ae0b40b12a23da54) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":166,"lbm_reads_lt_1ms":4}
I20240501 08:15:54.462852 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:54.494343 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:54.522292 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:54.539898 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:54.575459 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:54.612993 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:54.618391 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:54.633447 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:54.661178 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:54.696266 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:54.733686 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:54.748579 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:54.771672 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:54.806402 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:54.847349 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:54.858325 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:54.858729 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:54.951203 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:54.976030 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:15:54.976365 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:15:55.040446 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:55.099267 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling FlushMRSOp(4db7f887af8f425c8fa661196add4d37): perf score=0.033359
W20240501 08:15:55.122207 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:15:55.125695 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:15:55.166097 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(a9e70954aa4e4fcfbad12e8bfe3e0059): perf score=0.033376
I20240501 08:15:55.168551 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling FlushMRSOp(193de3c11ae54408a64ae3007bf88920): perf score=0.033369
W20240501 08:15:55.210676 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:55.211048 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:55.245326 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: FlushMRSOp(4db7f887af8f425c8fa661196add4d37) complete. Timing: real 0.145s user 0.072s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":2760,"dirs.queue_time_us":8493,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":9573,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":958,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17906,"threads_started":3,"wal-append.queue_time_us":6542}
I20240501 08:15:55.247331 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(4db7f887af8f425c8fa661196add4d37): 4834 bytes on disk
I20240501 08:15:55.248355 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(4db7f887af8f425c8fa661196add4d37) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240501 08:15:55.314594 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: FlushMRSOp(193de3c11ae54408a64ae3007bf88920) complete. Timing: real 0.145s user 0.055s sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":67302,"compiler_manager_pool.run_cpu_time_us":27,"compiler_manager_pool.run_wall_time_us":28,"dirs.queue_time_us":31750,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":10207,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":787,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":35117,"threads_started":2,"wal-append.queue_time_us":3355}
I20240501 08:15:55.316116 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(193de3c11ae54408a64ae3007bf88920): 4582 bytes on disk
I20240501 08:15:55.316856 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(193de3c11ae54408a64ae3007bf88920) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240501 08:15:55.334542 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(a9e70954aa4e4fcfbad12e8bfe3e0059) complete. Timing: real 0.168s user 0.057s sys 0.043s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":42517,"dirs.queue_time_us":10059,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":1451,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":964,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":59215,"threads_started":3,"wal-append.queue_time_us":6725}
I20240501 08:15:55.336668 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(193de3c11ae54408a64ae3007bf88920): perf score=0.033418
W20240501 08:15:55.344893 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:15:55.346606 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:55.395670 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(193de3c11ae54408a64ae3007bf88920) complete. Timing: real 0.059s user 0.040s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":612,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1465,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6909,"threads_started":1,"wal-append.queue_time_us":6783}
I20240501 08:15:55.397179 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(a9e70954aa4e4fcfbad12e8bfe3e0059): 4922 bytes on disk
I20240501 08:15:55.398197 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(a9e70954aa4e4fcfbad12e8bfe3e0059) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240501 08:15:55.399492 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(193de3c11ae54408a64ae3007bf88920): 4582 bytes on disk
I20240501 08:15:55.400132 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(193de3c11ae54408a64ae3007bf88920) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240501 08:15:55.467747 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:15:55.532282 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:15:55.589368 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:55.625821 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:55.641144 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:15:55.665786 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:55.678828 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:55.727944 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:55.728320 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:55.769209 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:55.770920 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:55.783383 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:55.783439 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:55.806375 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:55.822070 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:55.835414 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:55.853932 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:55.879330 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:55.885615 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:55.909767 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:15:55.980971 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:56.266283 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:56.303812 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:56.312155 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:56.318985 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:56.325549 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:56.352097 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:56.358740 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:56.389817 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:56.415570 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:15:56.614193 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:56.638947 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:56.641372 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:56.656809 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:56.696064 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:15:56.715921 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:56.724745 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:56.730175 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:56.734931 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:56.771641 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:56.782860 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:56.813163 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:56.845500 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:56.876549 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:56.879606 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:56.881078 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:56.918785 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:56.938771 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:56.942416 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:57.116477 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:57.161917 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:57.164060 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:57.170243 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:57.172348 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:57.212581 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:57.240907 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:57.257494 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling FlushMRSOp(78d1df7b0ea44f89b9f22c156f907c1e): perf score=0.033341
W20240501 08:15:57.320361 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:57.326714 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling FlushMRSOp(fdaad8e1d86942ddbc97958dcf31b344): perf score=0.033363
I20240501 08:15:57.332677 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(78d1df7b0ea44f89b9f22c156f907c1e): perf score=0.033365
W20240501 08:15:57.333468 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:15:57.366732 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:15:57.376163 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:57.387884 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: FlushMRSOp(78d1df7b0ea44f89b9f22c156f907c1e) complete. Timing: real 0.130s user 0.071s sys 0.027s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1075,"dirs.run_cpu_time_us":309,"dirs.run_wall_time_us":5027,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":810,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":185728,"thread_start_us":33018,"threads_started":2,"wal-append.queue_time_us":31468}
I20240501 08:15:57.389513 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(78d1df7b0ea44f89b9f22c156f907c1e): 4582 bytes on disk
I20240501 08:15:57.391381 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(78d1df7b0ea44f89b9f22c156f907c1e) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":256,"lbm_reads_lt_1ms":4}
W20240501 08:15:57.418406 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:57.452306 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: FlushMRSOp(fdaad8e1d86942ddbc97958dcf31b344) complete. Timing: real 0.125s user 0.036s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1497,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":1713,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1176,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7828,"threads_started":2,"wal-append.queue_time_us":6335}
I20240501 08:15:57.461634 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(78d1df7b0ea44f89b9f22c156f907c1e) complete. Timing: real 0.129s user 0.061s sys 0.033s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":13754,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":2147,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":901,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":58448,"threads_started":2,"wal-append.queue_time_us":44884}
I20240501 08:15:57.463310 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(78d1df7b0ea44f89b9f22c156f907c1e): 4582 bytes on disk
I20240501 08:15:57.464162 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(78d1df7b0ea44f89b9f22c156f907c1e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
I20240501 08:15:57.491957 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(fdaad8e1d86942ddbc97958dcf31b344): 4833 bytes on disk
I20240501 08:15:57.493060 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(fdaad8e1d86942ddbc97958dcf31b344) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 08:15:57.569352 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:57.574829 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:57.583698 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:57.584841 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:57.623066 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:57.624413 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:57.649549 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:57.661777 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:15:57.692112 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:15:57.747355 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:57.791198 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:57.795315 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:57.809572 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:57.810150 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:57.838893 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:57.839726 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:57.856774 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:57.869410 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:15:57.996114 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:15:58.097271 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:15:58.154672 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:58.157749 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:58.162410 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:58.171594 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:58.179976 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:58.182425 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:58.206708 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:58.214594 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:58.244675 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:58.261144 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:15:58.293196 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:58.300019 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:58.312072 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:58.327877 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:58.359953 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:15:58.373494 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:15:58.530802 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:58.539642 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:58.542649 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:58.569464 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:58.575454 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:58.575536 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:58.589974 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:58.647024 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling FlushMRSOp(4a6efa669c1645aaa971d59e5d38c8d5): perf score=0.033396
I20240501 08:15:58.665746 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(4a6efa669c1645aaa971d59e5d38c8d5): perf score=0.033393
I20240501 08:15:58.708149 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: FlushMRSOp(4a6efa669c1645aaa971d59e5d38c8d5) complete. Timing: real 0.061s user 0.045s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":401,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1642,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2416,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6492,"threads_started":2,"wal-append.queue_time_us":6370}
I20240501 08:15:58.709635 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(4a6efa669c1645aaa971d59e5d38c8d5): 4834 bytes on disk
I20240501 08:15:58.710376 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(4a6efa669c1645aaa971d59e5d38c8d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 08:15:58.711483 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling FlushMRSOp(358b6fe9c68349cb8cb963bb3a9ed2e9): perf score=0.033406
I20240501 08:15:58.729103 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(4a6efa669c1645aaa971d59e5d38c8d5) complete. Timing: real 0.063s user 0.031s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1825,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1212,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":892,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2345,"threads_started":2,"wal-append.queue_time_us":333}
I20240501 08:15:58.730515 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(4a6efa669c1645aaa971d59e5d38c8d5): 4834 bytes on disk
I20240501 08:15:58.731226 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(4a6efa669c1645aaa971d59e5d38c8d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20240501 08:15:58.763464 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:58.792313 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: FlushMRSOp(358b6fe9c68349cb8cb963bb3a9ed2e9) complete. Timing: real 0.081s user 0.056s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":1561,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":878,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5590,"threads_started":1,"wal-append.queue_time_us":5701}
I20240501 08:15:58.793386 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(358b6fe9c68349cb8cb963bb3a9ed2e9): 4921 bytes on disk
I20240501 08:15:58.794004 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(358b6fe9c68349cb8cb963bb3a9ed2e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 08:15:58.842612 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:58.847250 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:58.857437 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:15:58.858485 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:58.872165 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:58.874281 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:58.904413 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:58.958205 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:15:59.081593 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:15:59.086616 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:15:59.130941 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:15:59.144300 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:15:59.152941 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:59.156498 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:59.158303 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:59.167348 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:59.172389 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:59.177923 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:15:59.179752 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:15:59.219261 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:15:59.342583 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:59.356456 25594 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:15:59.356678 25594 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:15:59.357342 25594 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:15:59.360324 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:15:59.364514 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:59.364676 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:15:59.365701 25594 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:15:59.401706 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:59.434412 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:59.455533 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:59.467193 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:59.469429 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:59.481280 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:59.485363 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(dc19d80bad46460087c143778a4b0638): perf score=0.033340
I20240501 08:15:59.502312 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling FlushMRSOp(a7a56c284444496b8579600a49f19d1d): perf score=0.033339
W20240501 08:15:59.521044 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:15:59.521831 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:59.525993 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:59.528025 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:15:59.551108 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:59.553431 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:59.577785 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:59.640985 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:15:59.673015 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:59.723785 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(399d1a392b3f4d9e8531b6e0b5ad2a61): perf score=0.033402
W20240501 08:15:59.731326 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:59.732076 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(dc19d80bad46460087c143778a4b0638) complete. Timing: real 0.246s user 0.061s sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":28296,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1572,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":944,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":48328,"threads_started":2,"wal-append.queue_time_us":20019}
I20240501 08:15:59.733577 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(dc19d80bad46460087c143778a4b0638): 4581 bytes on disk
I20240501 08:15:59.734320 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(dc19d80bad46460087c143778a4b0638) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240501 08:15:59.739001 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: FlushMRSOp(a7a56c284444496b8579600a49f19d1d) complete. Timing: real 0.236s user 0.062s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8495,"dirs.run_cpu_time_us":300,"dirs.run_wall_time_us":13294,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":950,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":14512,"threads_started":2,"wal-append.queue_time_us":5988}
I20240501 08:15:59.740566 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d): 4834 bytes on disk
I20240501 08:15:59.741248 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240501 08:15:59.789525 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(399d1a392b3f4d9e8531b6e0b5ad2a61) complete. Timing: real 0.065s user 0.039s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":663,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":2695,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":787,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":474,"threads_started":2,"wal-append.queue_time_us":391}
I20240501 08:15:59.791002 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(399d1a392b3f4d9e8531b6e0b5ad2a61): 4921 bytes on disk
I20240501 08:15:59.791680 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(399d1a392b3f4d9e8531b6e0b5ad2a61) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240501 08:15:59.792457 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:15:59.792739 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(a7a56c284444496b8579600a49f19d1d): perf score=0.033418
I20240501 08:15:59.833854 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(a7a56c284444496b8579600a49f19d1d) complete. Timing: real 0.041s user 0.033s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1366,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":945,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":239,"threads_started":1,"wal-append.queue_time_us":432}
I20240501 08:15:59.835336 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d): 4834 bytes on disk
I20240501 08:15:59.836009 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240501 08:15:59.837447 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:15:59.884001 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:59.892663 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:59.896045 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:15:59.901701 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:15:59.928483 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:15:59.936698 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:15:59.946569 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:15:59.959638 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:15:59.993584 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:15:59.994908 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:16:00.148413 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:16:00.171448 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:16:00.177613 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:00.195497 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:00.207284 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:00.219038 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:00.224647 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:00.236171 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:00.265123 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:16:00.317166 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:16:00.363776 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:00.370237 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:00.386703 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:00.387718 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:00.392360 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:00.415190 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:00.419299 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:16:00.472338 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:00.494423 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:16:00.499869 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:16:00.541857 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:00.664531 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:00.678427 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:00.683998 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:00.715250 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:00.727090 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:00.739059 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(e9629fbbb4554edc9ba341d42b51f252): perf score=0.033380
I20240501 08:16:00.746616 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling FlushMRSOp(37a6157d7d734a80900139944481928c): perf score=0.033387
I20240501 08:16:00.757755 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:00.765558 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:16:00.784973 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:00.801954 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling FlushMRSOp(9b9bd7500e8747c6b8ee94a1faa1c5f7): perf score=0.033402
W20240501 08:16:00.807811 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:16:00.850970 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:00.933869 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: FlushMRSOp(9b9bd7500e8747c6b8ee94a1faa1c5f7) complete. Timing: real 0.132s user 0.045s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10767,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":2739,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":841,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":26676,"threads_started":2,"wal-append.queue_time_us":16169}
I20240501 08:16:00.935282 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(9b9bd7500e8747c6b8ee94a1faa1c5f7): 4922 bytes on disk
I20240501 08:16:00.936079 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(9b9bd7500e8747c6b8ee94a1faa1c5f7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 08:16:00.937091 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling FlushMRSOp(37a6157d7d734a80900139944481928c): perf score=0.033430
I20240501 08:16:00.956697 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(e9629fbbb4554edc9ba341d42b51f252) complete. Timing: real 0.217s user 0.052s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":17996,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":2170,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1000,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":36577,"threads_started":2,"wal-append.queue_time_us":16209}
I20240501 08:16:00.958469 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(9b9bd7500e8747c6b8ee94a1faa1c5f7): perf score=0.033430
I20240501 08:16:00.971010 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: FlushMRSOp(37a6157d7d734a80900139944481928c) complete. Timing: real 0.224s user 0.096s sys 0.048s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":58187,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1724,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":866,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":78252,"threads_started":2,"wal-append.queue_time_us":20264}
I20240501 08:16:00.972282 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(37a6157d7d734a80900139944481928c): 4834 bytes on disk
I20240501 08:16:00.980265 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(37a6157d7d734a80900139944481928c) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 08:16:00.981802 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:01.036964 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:01.050180 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: FlushMRSOp(37a6157d7d734a80900139944481928c) complete. Timing: real 0.112s user 0.047s sys 0.034s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1251,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":931,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":60750,"threads_started":1,"wal-append.queue_time_us":60532}
I20240501 08:16:01.072386 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(37a6157d7d734a80900139944481928c): 4834 bytes on disk
I20240501 08:16:01.073344 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(37a6157d7d734a80900139944481928c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240501 08:16:01.078457 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:01.082051 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:01.101714 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(9b9bd7500e8747c6b8ee94a1faa1c5f7) complete. Timing: real 0.140s user 0.075s sys 0.037s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":301,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":2571,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":992,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71492,"threads_started":1,"wal-append.queue_time_us":71597}
I20240501 08:16:01.103071 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(9b9bd7500e8747c6b8ee94a1faa1c5f7): 4922 bytes on disk
I20240501 08:16:01.105224 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(9b9bd7500e8747c6b8ee94a1faa1c5f7) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20240501 08:16:01.107317 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(e9629fbbb4554edc9ba341d42b51f252): 4581 bytes on disk
I20240501 08:16:01.108158 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(e9629fbbb4554edc9ba341d42b51f252) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 08:16:01.118661 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:01.121148 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:01.136135 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:16:01.274596 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:01.290669 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:01.291810 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:01.292248 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:01.292582 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:01.297142 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:01.378520 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:16:01.385390 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:16:01.462776 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:01.517925 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:01.579924 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling FlushMRSOp(491674dfbf74498e8a787bebaa5e95da): perf score=0.033333
I20240501 08:16:01.580220 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:01.586889 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:01.596014 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(6575b94bd51546f1bd3bb862f012e7ca): perf score=0.033338
I20240501 08:16:01.606570 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:01.612447 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(45d076303ea24d7fa0fed5a9d132019c): perf score=0.033340
I20240501 08:16:01.627872 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:01.635421 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:16:01.651907 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:01.661201 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:01.666577 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:16:01.693364 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:01.705395 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:01.774361 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:01.780926 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:16:01.793380 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:01.797834 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:01.852016 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:01.853843 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(6575b94bd51546f1bd3bb862f012e7ca) complete. Timing: real 0.251s user 0.047s sys 0.040s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":16259,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":2921,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":919,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":108500,"threads_started":2,"wal-append.queue_time_us":92444}
I20240501 08:16:01.855190 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(6575b94bd51546f1bd3bb862f012e7ca): 4581 bytes on disk
I20240501 08:16:01.857626 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(6575b94bd51546f1bd3bb862f012e7ca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
I20240501 08:16:01.860556 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:01.874819 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: FlushMRSOp(491674dfbf74498e8a787bebaa5e95da) complete. Timing: real 0.294s user 0.043s sys 0.061s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":117069,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":6912,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11141,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":123682,"threads_started":2,"wal-append.queue_time_us":6696}
I20240501 08:16:01.876082 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling FlushMRSOp(45d076303ea24d7fa0fed5a9d132019c): perf score=0.033412
I20240501 08:16:01.876883 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(45d076303ea24d7fa0fed5a9d132019c) complete. Timing: real 0.263s user 0.043s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":22783,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":10436,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":29242,"threads_started":2,"wal-append.queue_time_us":3300}
I20240501 08:16:01.878090 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(45d076303ea24d7fa0fed5a9d132019c): 4834 bytes on disk
I20240501 08:16:01.878755 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(45d076303ea24d7fa0fed5a9d132019c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
I20240501 08:16:01.883021 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:01.896373 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:01.902961 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:01.919184 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:16:01.933015 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: FlushMRSOp(45d076303ea24d7fa0fed5a9d132019c) complete. Timing: real 0.057s user 0.033s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1510,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":803,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":253,"threads_started":1,"wal-append.queue_time_us":357}
I20240501 08:16:01.934625 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(491674dfbf74498e8a787bebaa5e95da): 4921 bytes on disk
I20240501 08:16:01.935258 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(491674dfbf74498e8a787bebaa5e95da) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240501 08:16:01.936249 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(45d076303ea24d7fa0fed5a9d132019c): 4834 bytes on disk
I20240501 08:16:01.936772 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(45d076303ea24d7fa0fed5a9d132019c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240501 08:16:01.942507 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:02.039440 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:16:02.192708 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:02.211314 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:02.212010 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:02.230010 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:02.230159 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:02.239142 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:02.284750 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:02.453675 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:02.469448 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:02.492239 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:02.501212 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:02.517491 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:16:02.527365 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:16:02.528759 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:02.553344 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:02.562059 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:02.567797 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:02.680366 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:02.704084 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:02.710341 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:02.715984 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:02.744760 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:16:02.751618 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:02.759548 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:02.766963 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:02.803123 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:02.803901 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:16:02.819664 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:16:02.862651 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(658f2a172a2d47c8bf8b44c9d51132e4): perf score=0.033342
I20240501 08:16:02.885267 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(60238162349647c681685349c6f8ede4): perf score=0.033352
I20240501 08:16:02.943564 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(658f2a172a2d47c8bf8b44c9d51132e4) complete. Timing: real 0.081s user 0.057s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4193,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1357,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":864,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":1280,"thread_start_us":8544,"threads_started":2,"wal-append.queue_time_us":333}
I20240501 08:16:02.944849 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(658f2a172a2d47c8bf8b44c9d51132e4): 4921 bytes on disk
I20240501 08:16:02.945624 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(658f2a172a2d47c8bf8b44c9d51132e4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240501 08:16:02.978188 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(60238162349647c681685349c6f8ede4) complete. Timing: real 0.093s user 0.053s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":12755,"dirs.run_cpu_time_us":278,"dirs.run_wall_time_us":1507,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":825,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":20325,"threads_started":2,"wal-append.queue_time_us":1650}
I20240501 08:16:02.979513 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(60238162349647c681685349c6f8ede4): 4834 bytes on disk
I20240501 08:16:02.980508 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(60238162349647c681685349c6f8ede4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 08:16:02.988229 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling FlushMRSOp(f93c2d0f9c7947bb83a50c7fb8c3518a): perf score=0.033382
W20240501 08:16:03.106036 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:16:03.116137 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:03.123620 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:03.139163 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:03.156800 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:03.206638 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: FlushMRSOp(f93c2d0f9c7947bb83a50c7fb8c3518a) complete. Timing: real 0.218s user 0.092s sys 0.070s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6362,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":10162,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":873,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":99429,"threads_started":2,"wal-append.queue_time_us":93037}
I20240501 08:16:03.208438 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(f93c2d0f9c7947bb83a50c7fb8c3518a): 4581 bytes on disk
I20240501 08:16:03.210824 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(f93c2d0f9c7947bb83a50c7fb8c3518a) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240501 08:16:03.218125 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:03.254318 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:03.260659 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:03.309791 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:03.343006 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:03.352254 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:03.356706 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:03.357584 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:16:03.386895 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:03.392407 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:03.407222 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:03.485978 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:03.622676 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:03.630002 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:03.634888 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:03.658478 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:03.674461 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:03.696383 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:03.697615 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:03.761454 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:03.825853 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:03.848580 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:03.892228 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:03.938222 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:03.943430 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:03.947398 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling FlushMRSOp(13b4e2d5fe4846e8b3c6081abb3c53da): perf score=0.033357
I20240501 08:16:03.949962 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(d7903106d7e74e1ea102b1abdbfce63e): perf score=0.033359
I20240501 08:16:03.965622 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling FlushMRSOp(d7903106d7e74e1ea102b1abdbfce63e): perf score=0.033365
I20240501 08:16:03.965678 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:03.979311 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:03.985687 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(a25e1ddb987b4f8c8a4031053f581c7c): perf score=0.033370
I20240501 08:16:03.987689 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:04.014633 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:04.094381 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:04.113314 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:04.138391 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:04.145095 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: FlushMRSOp(13b4e2d5fe4846e8b3c6081abb3c53da) complete. Timing: real 0.197s user 0.033s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":24823,"dirs.run_cpu_time_us":315,"dirs.run_wall_time_us":18011,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":946,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":42063,"threads_started":2,"wal-append.queue_time_us":13686}
I20240501 08:16:04.150743 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(13b4e2d5fe4846e8b3c6081abb3c53da): 4921 bytes on disk
I20240501 08:16:04.151437 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(13b4e2d5fe4846e8b3c6081abb3c53da) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 08:16:04.160763 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(d7903106d7e74e1ea102b1abdbfce63e) complete. Timing: real 0.210s user 0.053s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":19854,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":13800,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":797,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67052,"threads_started":2,"wal-append.queue_time_us":33013}
I20240501 08:16:04.162142 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(d7903106d7e74e1ea102b1abdbfce63e): 4834 bytes on disk
I20240501 08:16:04.162834 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(d7903106d7e74e1ea102b1abdbfce63e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 08:16:04.163916 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(13b4e2d5fe4846e8b3c6081abb3c53da): perf score=0.033413
W20240501 08:16:04.195523 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:04.218685 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:04.219352 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:04.241576 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:04.255956 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:04.261961 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: FlushMRSOp(d7903106d7e74e1ea102b1abdbfce63e) complete. Timing: real 0.296s user 0.038s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":9212,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":1532,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":911,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":45315,"threads_started":2,"wal-append.queue_time_us":34515}
I20240501 08:16:04.263347 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(d7903106d7e74e1ea102b1abdbfce63e): 4834 bytes on disk
I20240501 08:16:04.264161 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(d7903106d7e74e1ea102b1abdbfce63e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240501 08:16:04.281790 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(13b4e2d5fe4846e8b3c6081abb3c53da) complete. Timing: real 0.118s user 0.049s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1589,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1498,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":910,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":28022,"threads_started":1,"wal-append.queue_time_us":14119}
I20240501 08:16:04.285425 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(13b4e2d5fe4846e8b3c6081abb3c53da): 4921 bytes on disk
I20240501 08:16:04.286415 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(13b4e2d5fe4846e8b3c6081abb3c53da) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240501 08:16:04.291651 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:04.295059 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:04.299042 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:04.308207 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(a25e1ddb987b4f8c8a4031053f581c7c) complete. Timing: real 0.322s user 0.053s sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39780,"dirs.run_cpu_time_us":278,"dirs.run_wall_time_us":17894,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":963,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":47599,"threads_started":2,"wal-append.queue_time_us":2758}
I20240501 08:16:04.309546 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(a25e1ddb987b4f8c8a4031053f581c7c): 4581 bytes on disk
I20240501 08:16:04.310169 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(a25e1ddb987b4f8c8a4031053f581c7c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240501 08:16:04.320622 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:04.352036 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:04.355731 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:04.362020 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:04.408632 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:04.530992 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:04.531060 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:04.554778 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:04.571223 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:04.604878 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:04.614895 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:04.740590 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:04.793781 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:04.796823 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:04.811369 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:04.853193 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:04.864439 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:04.894886 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:04.895231 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:04.961203 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:04.988027 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:05.012627 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:05.018450 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling FlushMRSOp(29506d751571401f86bced7e6f5415d7): perf score=0.033358
I20240501 08:16:05.030555 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:05.041744 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:05.044021 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:05.044945 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(cfc0c7b429754d92b4dd166c073ac8ed): perf score=0.033361
W20240501 08:16:05.059269 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:05.067247 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:05.070613 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(cfc0c7b429754d92b4dd166c073ac8ed): perf score=0.033369
I20240501 08:16:05.131516 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:05.133244 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: FlushMRSOp(29506d751571401f86bced7e6f5415d7) complete. Timing: real 0.114s user 0.043s sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":18285,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":1555,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":729,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":24526,"threads_started":2,"wal-append.queue_time_us":6484}
I20240501 08:16:05.134680 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(29506d751571401f86bced7e6f5415d7): 4834 bytes on disk
I20240501 08:16:05.135799 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(29506d751571401f86bced7e6f5415d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 08:16:05.156877 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling FlushMRSOp(29506d751571401f86bced7e6f5415d7): perf score=0.033391
I20240501 08:16:05.166311 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:05.174157 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:05.187628 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:05.207000 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:16:05.209059 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:16:05.224047 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(cfc0c7b429754d92b4dd166c073ac8ed) complete. Timing: real 0.153s user 0.046s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":16122,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":15591,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":833,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":17338,"threads_started":2,"wal-append.queue_time_us":1189}
I20240501 08:16:05.225636 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(cfc0c7b429754d92b4dd166c073ac8ed): 4921 bytes on disk
I20240501 08:16:05.226697 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(cfc0c7b429754d92b4dd166c073ac8ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 08:16:05.231810 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(cfc0c7b429754d92b4dd166c073ac8ed) complete. Timing: real 0.187s user 0.052s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12675,"dirs.run_cpu_time_us":333,"dirs.run_wall_time_us":13593,"drs_written":1,"lbm_read_time_us":339,"lbm_reads_lt_1ms":4,"lbm_write_time_us":943,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12706,"threads_started":2,"wal-append.queue_time_us":8003}
W20240501 08:16:05.232983 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 08:16:05.237075 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(cfc0c7b429754d92b4dd166c073ac8ed): 4921 bytes on disk
I20240501 08:16:05.239590 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(cfc0c7b429754d92b4dd166c073ac8ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240501 08:16:05.255126 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: FlushMRSOp(29506d751571401f86bced7e6f5415d7) complete. Timing: real 0.098s user 0.053s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":17277,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1578,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7998,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":31144,"threads_started":2,"wal-append.queue_time_us":14099}
I20240501 08:16:05.256686 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(29506d751571401f86bced7e6f5415d7): 4834 bytes on disk
I20240501 08:16:05.257323 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(29506d751571401f86bced7e6f5415d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240501 08:16:05.364326 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:05.367182 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:05.376534 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:05.383312 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:05.399277 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:05.407196 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:05.429280 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:05.497844 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:16:05.532195 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:05.558180 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:16:05.589112 27277 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:05.589316 27277 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:05.590014 27277 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:16:05.592445 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:16:05.594373 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:05.594530 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:05.595548 27277 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:16:05.617772 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:05.626735 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:05.648416 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:05.656323 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:05.670835 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:05.680145 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:05.859251 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:16:05.933118 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:05.949391 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:05.950486 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:05.952417 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:05.966660 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:05.995708 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:06.007280 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:06.030578 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:06.032024 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:06.033372 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:06.033530 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:06.076974 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 52631.5us
W20240501 08:16:06.093405 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:06.106225 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:06.135546 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:06.182869 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:06.190528 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:06.198413 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:06.218400 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:06.380745 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:06.404563 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:06.423285 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:06.432559 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:06.434780 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:06.436621 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:06.446869 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:06.451864 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:06.490172 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:06.540724 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:06.597513 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:06.621244 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:06.623548 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:06.718318 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:06.763499 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:06.792829 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:06.829751 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:06.837632 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:06.858282 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:06.861358 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:06.873795 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:06.880506 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:06.889348 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:06.891633 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:06.904546 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:06.917193 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:06.968905 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:07.040150 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:07.059963 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:07.070462 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:07.120553 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:07.185669 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:07.188315 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:07.188743 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:07.198122 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:07.199669 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:07.209606 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:07.318506 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:07.337455 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:07.377103 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:07.412578 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:07.448429 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:07.462742 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:07.470609 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:07.512413 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:07.513502 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:07.591658 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:07.598498 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:07.604096 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:07.605196 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:07.620018 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:07.626500 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:07.649156 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:07.671449 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:07.700018 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:07.707732 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:07.745461 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:07.786850 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:16:07.839486 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:07.850126 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:07.860239 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:07.875095 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:07.894837 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:07.903072 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:07.988142 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:08.134954 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:08.142220 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:08.151309 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:08.158982 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:08.162217 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:08.164886 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:08.208263 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:08.321036 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:08.339246 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:16:08.388840 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:08.395243 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:08.397405 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:08.406006 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:08.407240 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:08.424296 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:08.431624 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:08.445225 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:08.492897 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:08.570123 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:08.644948 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:08.675946 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:08.687132 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:08.700788 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:08.727489 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:08.729631 27422 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:08.729835 27422 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:08.737970 27422 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:16:08.745251 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240501 08:16:08.749761 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:08.758423 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:08.781117 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:08.790692 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:08.798094 8895 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.107.5:45691, user_credentials={real_user=slave}} blocked reactor thread for 52618.8us
W20240501 08:16:08.822088 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:16:08.832172 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:08.832437 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:08.832511 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:08.832984 27422 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:16:08.843742 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:08.875555 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:16:08.893256 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:08.928251 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:08.942776 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:08.944564 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:08.989351 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:09.031212 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:09.060801 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:09.166796 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:09.182277 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:09.185616 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:09.201231 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:09.207198 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:09.216511 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:09.270787 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:09.408906 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:09.443205 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:09.452414 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:09.470798 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:09.490514 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:09.507606 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:09.531277 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:09.536015 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:09.552160 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:16:09.576238 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:09.577749 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:09.598887 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:09.630112 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:09.689126 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:09.704411 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:09.720131 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:09.766845 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:09.828051 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:16:09.905568 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:09.906720 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:09.933595 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:09.945801 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:09.959270 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:09.963701 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:09.966571 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:09.978847 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:10.051862 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:10.097695 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:10.115391 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:10.141813 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:10.207751 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:10.209365 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:10.231153 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:10.236117 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:10.248659 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:10.262915 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:10.274513 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:10.294867 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:16:10.296205 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:16:10.313944 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:10.352717 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 08:16:10.393172 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:10.418170 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:10.438418 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:10.438915 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:10.463367 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:10.486743 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:10.490048 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:10.577980 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:16:10.593150 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:16:10.594118 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:10.665994 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:16:10.697307 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:10.705159 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:10.723616 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:10.723811 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:10.745807 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:10.750770 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:10.820840 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:10.901638 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:10.921130 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:10.926199 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:10.941352 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:10.946231 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:10.965587 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:10.980398 27639 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:10.980650 27639 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:10.981494 27639 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:16:10.984684 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:16:10.985205 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:10.985332 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:10.985860 27639 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:16:11.043816 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:11.048821 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:11.216100 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:11.226729 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:11.228281 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:11.234323 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:11.243083 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:11.255249 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:11.279354 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:11.282835 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:11.286829 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:11.332971 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:11.365737 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:16:11.479043 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:11.489046 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:11.491755 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:11.506067 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:11.541635 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:16:11.548609 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:11.555795 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:11.555851 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:11.588076 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:11.588387 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 67740.8us
W20240501 08:16:11.591449 6835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 50544.7us
I20240501 08:16:11.673666 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:11.697036 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:11.699286 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:11.727166 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:11.735248 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:11.764120 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:11.767712 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:11.847797 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:11.879092 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:11.936923 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:16:12.009269 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:12.024955 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:12.028734 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:12.048079 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:12.051613 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:12.070945 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:12.074308 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:12.119619 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:12.144438 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:12.207984 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:12.227286 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:12.249752 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:12.250901 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:12.285182 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:12.293599 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:12.295965 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:12.408068 27422 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:12.408259 27422 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:12.408947 27422 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:16:12.428354 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:12.428575 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:16:12.428689 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:12.429049 27422 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:16:12.447793 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:12.472419 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:12.487879 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:12.491423 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:12.510922 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:12.514804 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:12.576617 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:12.593124 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:12.737740 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:12.741214 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:12.745921 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:16:12.792035 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:12.793639 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:12.829032 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:12.851152 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:12.880946 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:12.884233 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:12.891789 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:12.938594 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:16:12.950278 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:12.951761 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:12.975572 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:16:12.982587 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:13.000162 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:13.015086 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:16:13.015437 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:13.032714 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 51959.4us
W20240501 08:16:13.039233 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:13.053243 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:13.181330 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:13.192546 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:13.255812 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:13.256232 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:13.257158 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:13.266285 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:13.279048 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:13.302951 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:13.398530 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:13.497289 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:13.538162 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:13.556849 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:13.576699 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:13.590675 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:13.595995 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:13.600092 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:13.653285 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:13.667457 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:13.704139 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:13.718333 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:13.737203 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:13.760730 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:13.768249 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:13.789963 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:13.917595 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:13.969663 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:13.993714 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:14.022344 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:14.025305 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:14.033231 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:14.078229 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:14.149377 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:14.170406 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:14.171697 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:14.178411 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:14.180203 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:14.204063 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:14.206876 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:14.211669 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:14.357446 27422 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:14.357666 27422 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:14.358424 27422 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:16:14.360050 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 08:16:14.361624 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:14.361820 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:16:14.361932 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:14.362170 27422 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:16:14.384120 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:14.384496 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:14.409888 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:14.419121 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:14.420740 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:14.442044 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:14.445458 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:14.447839 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:14.595947 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:14.596530 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:14.607507 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:14.615378 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:14.632310 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:14.634940 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:14.644804 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:14.659968 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:14.876142 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:14.897455 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:14.956125 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:14.959439 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:14.974771 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:14.982663 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:14.999531 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:14.999812 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:15.119304 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:15.127002 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:15.129204 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:15.142555 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:15.151996 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:15.155129 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:15.181425 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:15.187394 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:15.197002 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:15.202052 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:15.223235 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(6575b94bd51546f1bd3bb862f012e7ca): perf score=0.033335
W20240501 08:16:15.255038 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:15.292387 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:16:15.317451 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(6575b94bd51546f1bd3bb862f012e7ca) complete. Timing: real 0.093s user 0.057s sys 0.029s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":21543,"dirs.queue_time_us":776,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":3174,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":961,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":22452,"threads_started":3,"wal-append.queue_time_us":512}
I20240501 08:16:15.319998 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(6575b94bd51546f1bd3bb862f012e7ca): 4581 bytes on disk
I20240501 08:16:15.321192 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(6575b94bd51546f1bd3bb862f012e7ca) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
W20240501 08:16:15.323110 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:16:15.323616 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(8b7f87b6032d45c5be5d748ab2cc71e2): perf score=0.033355
I20240501 08:16:15.371939 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(8b7f87b6032d45c5be5d748ab2cc71e2) complete. Timing: real 0.048s user 0.031s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1604,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":914,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":263,"threads_started":1,"wal-append.queue_time_us":386}
I20240501 08:16:15.373338 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(8b7f87b6032d45c5be5d748ab2cc71e2): 4921 bytes on disk
I20240501 08:16:15.373942 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(8b7f87b6032d45c5be5d748ab2cc71e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240501 08:16:15.419505 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:15.468029 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:15.471828 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240501 08:16:15.483690 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:15.532917 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:15.539181 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:15.591059 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:15.605741 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:15.610955 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:16:15.621699 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:16:15.650310 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:16:15.651679 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:15.677268 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:15.833621 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:15.837358 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:15.997691 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:16.028476 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:16.092490 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:16.149395 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:16.149995 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:16.167492 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:16.192360 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:16.211536 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:16.220958 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:16.252243 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:16.263321 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:16.316711 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:16.354719 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:16.359194 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:16.367501 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:16.370601 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:16.404815 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:16.422919 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:16.519779 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:16.524699 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:16.588171 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(13b4e2d5fe4846e8b3c6081abb3c53da): perf score=0.033337
I20240501 08:16:16.596076 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:16.600175 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:16.616864 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:16.622555 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:16.623629 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:16.634543 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:16.645031 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:16.678022 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:16.747390 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(13b4e2d5fe4846e8b3c6081abb3c53da) complete. Timing: real 0.158s user 0.059s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":2308,"dirs.queue_time_us":6407,"dirs.run_cpu_time_us":285,"dirs.run_wall_time_us":8074,"drs_written":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2004,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15664,"threads_started":3,"wal-append.queue_time_us":6557}
I20240501 08:16:16.749758 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(491674dfbf74498e8a787bebaa5e95da): perf score=0.033349
W20240501 08:16:16.825527 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:16.829891 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(491674dfbf74498e8a787bebaa5e95da) complete. Timing: real 0.080s user 0.040s sys 0.026s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1484,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1048,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11838,"threads_started":1,"wal-append.queue_time_us":11990}
I20240501 08:16:16.831553 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(4a6efa669c1645aaa971d59e5d38c8d5): perf score=0.033337
W20240501 08:16:16.893082 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:16.937042 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(4a6efa669c1645aaa971d59e5d38c8d5) complete. Timing: real 0.105s user 0.066s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1542,"drs_written":1,"lbm_read_time_us":314,"lbm_reads_lt_1ms":4,"lbm_write_time_us":802,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":14893,"threads_started":1,"wal-append.queue_time_us":14898}
I20240501 08:16:16.938956 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(13b4e2d5fe4846e8b3c6081abb3c53da): 4921 bytes on disk
I20240501 08:16:16.940687 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(13b4e2d5fe4846e8b3c6081abb3c53da) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240501 08:16:16.942373 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(491674dfbf74498e8a787bebaa5e95da): 4921 bytes on disk
I20240501 08:16:16.951066 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(491674dfbf74498e8a787bebaa5e95da) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":175,"lbm_reads_lt_1ms":4}
I20240501 08:16:16.952742 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(4a6efa669c1645aaa971d59e5d38c8d5): 4834 bytes on disk
W20240501 08:16:16.953454 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:16.953373 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(4a6efa669c1645aaa971d59e5d38c8d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240501 08:16:17.003309 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:17.099400 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:17.116948 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:17.133049 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(cfc0c7b429754d92b4dd166c073ac8ed): perf score=0.033380
I20240501 08:16:17.216765 28866 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:17.217017 28866 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:17.217849 28866 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:16:17.221117 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:17.261091 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:17.282187 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(cfc0c7b429754d92b4dd166c073ac8ed) complete. Timing: real 0.149s user 0.059s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12762,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":2539,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2046,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":52199,"threads_started":2,"wal-append.queue_time_us":39678}
I20240501 08:16:17.284418 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:17.285598 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(cfc0c7b429754d92b4dd166c073ac8ed): 4921 bytes on disk
I20240501 08:16:17.286885 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(cfc0c7b429754d92b4dd166c073ac8ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
W20240501 08:16:17.287312 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:16:17.287483 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:17.289387 28866 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:16:17.302564 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:17.324489 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:17.347955 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:17.368352 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:17.371096 6833 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 58922.4us
I20240501 08:16:17.375880 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:17.379801 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:17.400135 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:17.416836 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:17.419344 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:17.452533 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:17.462680 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:17.514235 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:17.580549 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(60238162349647c681685349c6f8ede4): perf score=0.033402
W20240501 08:16:17.597021 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:17.621148 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(60238162349647c681685349c6f8ede4) complete. Timing: real 0.040s user 0.030s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1566,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":633,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":363,"threads_started":1,"wal-append.queue_time_us":514}
I20240501 08:16:17.622606 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(60238162349647c681685349c6f8ede4): 4834 bytes on disk
I20240501 08:16:17.623277 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(60238162349647c681685349c6f8ede4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
W20240501 08:16:17.752898 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:17.764334 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:17.790146 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:17.815196 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:16:17.897908 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:17.930181 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:17.938189 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:16:17.978480 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:18.017782 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:18.031121 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:18.044000 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:18.049451 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:18.065562 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:18.067080 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:18.107825 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:18.125548 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:18.126684 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(29506d751571401f86bced7e6f5415d7): perf score=0.033399
I20240501 08:16:18.145673 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:18.154048 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:18.154434 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:18.156117 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:18.182657 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:18.210317 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(a9e70954aa4e4fcfbad12e8bfe3e0059): perf score=0.033387
W20240501 08:16:18.280925 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:18.302898 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(a9e70954aa4e4fcfbad12e8bfe3e0059) complete. Timing: real 0.092s user 0.046s sys 0.027s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":24694,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":10662,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":814,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":31143,"threads_started":2,"wal-append.queue_time_us":6376}
I20240501 08:16:18.305101 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(a9e70954aa4e4fcfbad12e8bfe3e0059): 4922 bytes on disk
I20240501 08:16:18.305819 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(a9e70954aa4e4fcfbad12e8bfe3e0059) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 08:16:18.308163 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(29506d751571401f86bced7e6f5415d7) complete. Timing: real 0.180s user 0.064s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":17406,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":4532,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":873,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":25539,"threads_started":2,"wal-append.queue_time_us":5459}
I20240501 08:16:18.310050 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(29506d751571401f86bced7e6f5415d7): 4834 bytes on disk
I20240501 08:16:18.310765 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(29506d751571401f86bced7e6f5415d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240501 08:16:18.431679 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:18.450635 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:18.471669 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:18.485790 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:18.487078 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:18.497550 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:18.510417 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:18.548787 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:18.570297 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:16:18.608610 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:18.750551 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:18.755964 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:18.777331 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:18.785378 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:18.809444 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:18.815019 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:18.849478 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 58489.2us
W20240501 08:16:18.888002 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:16:18.905442 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:18.920030 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:18.962152 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:19.009244 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:19.016557 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:19.033579 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:19.050809 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:19.065438 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(e9629fbbb4554edc9ba341d42b51f252): perf score=0.033370
I20240501 08:16:19.065907 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:19.100378 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:19.151489 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:19.172904 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(e9629fbbb4554edc9ba341d42b51f252) complete. Timing: real 0.107s user 0.061s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7314,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":2418,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":913,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7387,"threads_started":2,"wal-append.queue_time_us":405}
I20240501 08:16:19.174436 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(e9629fbbb4554edc9ba341d42b51f252): 4581 bytes on disk
I20240501 08:16:19.175194 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(e9629fbbb4554edc9ba341d42b51f252) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240501 08:16:19.176326 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(049090443eaa4b18ba90cb60af9a87ab): perf score=0.033394
W20240501 08:16:19.196760 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:19.221050 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(049090443eaa4b18ba90cb60af9a87ab) complete. Timing: real 0.044s user 0.032s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1435,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1117,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":308,"threads_started":1,"wal-append.queue_time_us":428}
I20240501 08:16:19.222226 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(049090443eaa4b18ba90cb60af9a87ab): 4581 bytes on disk
I20240501 08:16:19.222851 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(049090443eaa4b18ba90cb60af9a87ab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
W20240501 08:16:19.280040 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:19.298125 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:19.340914 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:19.347605 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:19.349777 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:19.371955 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:19.374846 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:19.391669 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:19.443804 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:19.465721 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:19.482769 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:19.509074 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:19.510392 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:19.538704 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:19.558084 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:19.570628 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:19.575831 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:19.576800 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:19.586918 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:19.736207 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:19.797044 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:19.815749 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:19.819629 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:19.820876 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:19.821385 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:19.824501 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(a25e1ddb987b4f8c8a4031053f581c7c): perf score=0.033401
I20240501 08:16:19.833014 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:19.924965 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:19.982522 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(a25e1ddb987b4f8c8a4031053f581c7c) complete. Timing: real 0.158s user 0.078s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8701,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1545,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":910,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":16339,"threads_started":2,"wal-append.queue_time_us":7835}
I20240501 08:16:19.984267 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(a25e1ddb987b4f8c8a4031053f581c7c): 4581 bytes on disk
I20240501 08:16:19.985050 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(a25e1ddb987b4f8c8a4031053f581c7c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
W20240501 08:16:20.030419 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:20.072041 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:16:20.075179 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:20.096398 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:20.100068 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:20.121297 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:20.142751 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:20.148694 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:20.150956 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:20.211761 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:20.227744 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(a7a56c284444496b8579600a49f19d1d): perf score=0.033400
I20240501 08:16:20.237608 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(13b4e2d5fe4846e8b3c6081abb3c53da): perf score=0.033382
I20240501 08:16:20.284777 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(a7a56c284444496b8579600a49f19d1d) complete. Timing: real 0.057s user 0.038s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5068,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2178,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":948,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":12198,"threads_started":2,"wal-append.queue_time_us":7124}
I20240501 08:16:20.288017 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d): 4834 bytes on disk
I20240501 08:16:20.288991 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 08:16:20.290565 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(5f090fda814f455a88dccd6680bdd05f): perf score=0.033339
I20240501 08:16:20.291639 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(13b4e2d5fe4846e8b3c6081abb3c53da) complete. Timing: real 0.054s user 0.048s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1406,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":786,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3882,"threads_started":1,"wal-append.queue_time_us":3766}
I20240501 08:16:20.292789 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(13b4e2d5fe4846e8b3c6081abb3c53da): 4921 bytes on disk
I20240501 08:16:20.293466 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(13b4e2d5fe4846e8b3c6081abb3c53da) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240501 08:16:20.301492 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:20.321548 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:20.332684 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:20.336433 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:20.347415 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:16:20.355979 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:20.358196 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:20.366498 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:20.375119 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:20.376575 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:20.387202 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(5f090fda814f455a88dccd6680bdd05f) complete. Timing: real 0.089s user 0.066s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1847,"drs_written":1,"lbm_read_time_us":152,"lbm_reads_lt_1ms":4,"lbm_write_time_us":829,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":18996,"threads_started":1,"wal-append.queue_time_us":19099}
I20240501 08:16:20.389231 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(5f090fda814f455a88dccd6680bdd05f): 4834 bytes on disk
I20240501 08:16:20.389978 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(5f090fda814f455a88dccd6680bdd05f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":188,"lbm_reads_lt_1ms":4}
W20240501 08:16:20.393623 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:16:20.403380 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:16:20.559504 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:16:20.610525 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:20.648766 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:20.660629 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:16:20.681775 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:20.702374 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:20.720697 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240501 08:16:20.721470 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:20.723425 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:20.734319 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:20.751785 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:20.759323 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:20.786556 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:20.795815 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:20.795993 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:16:20.805276 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:20.806151 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:20.848455 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:21.008576 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:21.096607 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:21.125454 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:21.224411 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:21.277953 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:21.298938 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(358b6fe9c68349cb8cb963bb3a9ed2e9): perf score=0.033391
I20240501 08:16:21.310668 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:21.316948 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:21.366592 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:21.373409 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:21.379772 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:21.401496 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:21.446173 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:21.483606 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:21.484042 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:21.499027 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:21.518338 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:21.532234 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:21.549777 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:21.562481 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:16:21.608775 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:21.626003 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(358b6fe9c68349cb8cb963bb3a9ed2e9) complete. Timing: real 0.327s user 0.047s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":27991,"dirs.run_cpu_time_us":299,"dirs.run_wall_time_us":24454,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4127,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10403,"threads_started":2,"wal-append.queue_time_us":317}
I20240501 08:16:21.627687 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(358b6fe9c68349cb8cb963bb3a9ed2e9): 4921 bytes on disk
I20240501 08:16:21.628494 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(358b6fe9c68349cb8cb963bb3a9ed2e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4}
I20240501 08:16:21.646652 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(399d1a392b3f4d9e8531b6e0b5ad2a61): perf score=0.033364
W20240501 08:16:21.665377 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:16:21.673683 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:21.711800 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(399d1a392b3f4d9e8531b6e0b5ad2a61) complete. Timing: real 0.065s user 0.040s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":387,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":1641,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":824,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":552,"threads_started":2,"wal-append.queue_time_us":406}
I20240501 08:16:21.713018 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(399d1a392b3f4d9e8531b6e0b5ad2a61): 4921 bytes on disk
I20240501 08:16:21.713593 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(399d1a392b3f4d9e8531b6e0b5ad2a61) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
W20240501 08:16:21.771245 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:21.780244 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:21.782162 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:21.802174 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:21.813840 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:21.819573 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:21.820482 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:21.820899 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:21.835314 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:16:21.899753 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:22.112815 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:22.154187 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:22.154927 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:22.156940 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:22.161973 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:22.189158 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:22.195703 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:22.203255 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:22.210453 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:22.225469 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:22.276088 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:22.305588 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:22.326012 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:22.356024 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:22.357843 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:22.411243 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:16:22.439148 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:16:22.468561 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:22.534463 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:22.540164 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:22.540722 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:22.546604 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:22.557766 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:22.567432 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:22.572181 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:22.602208 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:22.628470 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:16:22.732566 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:22.759495 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:16:22.838568 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:22.838703 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:22.842568 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:22.842720 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:22.856789 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:22.874871 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:22.886888 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:22.907779 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:23.007687 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:23.015800 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:23.036798 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:23.051580 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:23.053469 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:23.117074 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:23.134697 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:23.136955 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(9b9bd7500e8747c6b8ee94a1faa1c5f7): perf score=0.033397
I20240501 08:16:23.184944 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(9b9bd7500e8747c6b8ee94a1faa1c5f7) complete. Timing: real 0.048s user 0.030s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":383,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1509,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":659,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":503,"threads_started":2,"wal-append.queue_time_us":330}
I20240501 08:16:23.186106 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(9b9bd7500e8747c6b8ee94a1faa1c5f7): 4922 bytes on disk
I20240501 08:16:23.186956 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(9b9bd7500e8747c6b8ee94a1faa1c5f7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240501 08:16:23.316524 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:23.317498 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:23.344192 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:23.351137 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:23.356510 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:23.374943 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:23.384406 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:23.439265 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(dc19d80bad46460087c143778a4b0638): perf score=0.033385
I20240501 08:16:23.500798 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:23.536577 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:23.537550 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:23.540179 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:23.543381 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(dc19d80bad46460087c143778a4b0638) complete. Timing: real 0.104s user 0.044s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":2426,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":902,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17970,"threads_started":1,"wal-append.queue_time_us":17817}
I20240501 08:16:23.545159 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(dc19d80bad46460087c143778a4b0638): 4581 bytes on disk
I20240501 08:16:23.545802 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(dc19d80bad46460087c143778a4b0638) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240501 08:16:23.588263 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:23.594441 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:23.600533 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:23.630600 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:23.665431 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:16:23.676548 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:23.739307 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:23.775583 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:23.782866 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:23.784020 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:23.784027 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:23.790767 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:23.798096 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(658f2a172a2d47c8bf8b44c9d51132e4): perf score=0.033353
I20240501 08:16:23.798585 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:23.813633 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:23.916224 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(658f2a172a2d47c8bf8b44c9d51132e4) complete. Timing: real 0.118s user 0.048s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1559,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4792,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"mutex_wait_us":2,"peak_mem_usage":0,"rows_written":689,"thread_start_us":416,"threads_started":1,"wal-append.queue_time_us":372}
I20240501 08:16:23.917795 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(658f2a172a2d47c8bf8b44c9d51132e4): 4921 bytes on disk
I20240501 08:16:23.918571 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(658f2a172a2d47c8bf8b44c9d51132e4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":153,"lbm_reads_lt_1ms":4}
W20240501 08:16:23.994966 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:24.054416 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:24.059206 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:24.060570 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:24.085542 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:24.092247 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:24.107584 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:24.124511 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:24.137039 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:24.145306 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:24.145571 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:16:24.248422 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:16:24.279007 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:24.309227 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:24.309360 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:24.319543 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:24.333878 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:24.337590 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:24.344506 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:24.443168 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:24.458752 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:24.582161 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:24.596581 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:24.615038 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:24.638509 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:24.641640 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:24.657081 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:24.684034 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 70366.9us
I20240501 08:16:24.698765 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:24.701406 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:24.705782 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:24.747947 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:24.754360 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:24.754494 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:24.807047 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:24.807416 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:24.837745 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:24.842182 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:16:24.923429 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(37a6157d7d734a80900139944481928c): perf score=0.033335
I20240501 08:16:24.960511 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(37a6157d7d734a80900139944481928c) complete. Timing: real 0.037s user 0.031s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":1542,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":722,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":891,"threads_started":2,"wal-append.queue_time_us":791}
I20240501 08:16:24.962110 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(37a6157d7d734a80900139944481928c): 4834 bytes on disk
I20240501 08:16:24.962793 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(37a6157d7d734a80900139944481928c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240501 08:16:24.978912 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(60238162349647c681685349c6f8ede4): perf score=0.033340
I20240501 08:16:25.016358 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:25.032945 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:25.041059 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:25.066399 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:25.084841 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:25.095995 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:25.097157 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:25.102027 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(60238162349647c681685349c6f8ede4) complete. Timing: real 0.123s user 0.043s sys 0.051s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11464,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":1860,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":924,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":55513,"threads_started":2,"wal-append.queue_time_us":48298}
I20240501 08:16:25.104159 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(60238162349647c681685349c6f8ede4): 4834 bytes on disk
I20240501 08:16:25.105753 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(60238162349647c681685349c6f8ede4) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":384,"lbm_reads_lt_1ms":4}
I20240501 08:16:25.115309 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:25.124405 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:25.133863 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:25.145293 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:25.150421 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:25.229952 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:25.231655 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:16:25.362661 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:16:25.374145 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:25.390115 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:25.406069 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:25.409178 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:25.412019 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:25.430543 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:25.435503 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:25.593858 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:25.606238 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:25.612980 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:25.618089 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:25.635481 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:25.643695 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:25.651482 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:25.664261 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:16:25.727650 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:16:25.860922 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:16:25.884291 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:25.899955 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:25.902530 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:25.902657 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:25.903558 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:25.944779 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:26.019634 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:16:26.159971 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:26.164230 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:26.167404 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:26.197299 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:26.206594 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:26.208501 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:26.223657 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:26.248685 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:26.258414 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:26.260733 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:26.310299 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:16:26.316673 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:26.381989 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:26.394783 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:26.397006 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:26.398523 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:26.402568 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:26.435041 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:26.519416 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:26.557121 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:16:26.610205 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:16:26.638808 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:26.648962 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:26.664027 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:26.746836 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:26.750002 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:26.776540 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:26.792053 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:26.802042 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:26.840219 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:26.842728 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:26.848806 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:26.907419 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:26.907999 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:26.922247 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:26.959048 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:26.966027 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:26.991534 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:27.050349 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:16:27.193409 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:27.243247 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:27.246376 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:27.247190 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:27.266889 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:27.267597 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:27.279062 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:27.281854 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:27.288378 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:27.357528 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:27.439622 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:27.559892 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:27.578333 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:16:27.578724 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:27.583010 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:27.605727 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:27.612968 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:27.618448 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:27.624300 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:27.638152 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:27.656160 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:27.687745 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:27.694983 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:27.696413 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:27.741070 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:27.752993 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:27.764595 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:16:27.765576 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:27.769243 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:27.779078 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:28.015058 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:28.025815 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:28.028851 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:28.031675 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:28.034744 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:28.054378 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:28.081159 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:28.225801 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:28.304055 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:28.304200 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:28.325078 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:28.328114 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:28.334998 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:28.335124 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:28.349193 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:28.496749 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:28.508193 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:28.543588 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:28.549784 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:28.550714 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:28.589870 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:28.676013 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:28.679231 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:28.683661 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:28.708916 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:28.817153 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:28.818847 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:28.821638 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:28.843633 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:28.847932 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:28.851730 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:28.870605 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:28.878839 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:29.045154 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:29.064430 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:29.066691 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:29.081336 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:29.095986 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:29.110697 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:29.151779 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:29.155543 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:29.175839 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:29.177558 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:29.262015 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:29.361886 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:29.363966 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:29.372975 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:29.393988 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:29.395534 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:29.398806 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:29.406998 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:29.419090 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:29.472199 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:29.482198 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:29.581950 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling FlushMRSOp(4db7f887af8f425c8fa661196add4d37): perf score=0.033369
W20240501 08:16:29.637509 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:29.639173 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: FlushMRSOp(4db7f887af8f425c8fa661196add4d37) complete. Timing: real 0.054s user 0.040s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8890,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":5987,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1190,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":18656,"threads_started":2,"wal-append.queue_time_us":10863}
I20240501 08:16:29.641181 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(4db7f887af8f425c8fa661196add4d37): 4834 bytes on disk
I20240501 08:16:29.642122 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(4db7f887af8f425c8fa661196add4d37) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20240501 08:16:29.741235 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:29.742237 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:29.750331 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:29.755560 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:29.760622 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:29.769531 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:29.777704 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:29.787045 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:29.793869 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:29.810084 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:29.886159 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:29.890012 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:29.896544 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:29.916524 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:29.920679 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:29.922698 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:29.948894 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:29.963313 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:30.110489 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:30.346563 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:30.352162 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:30.364861 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:16:30.366339 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:16:30.371780 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:16:30.417253 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:30.456548 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:30.464013 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:30.481415 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:30.491783 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:30.493031 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:30.515543 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:30.677343 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:30.706221 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:30.707198 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:30.722673 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:30.773499 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240501 08:16:30.786794 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:30.791146 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:30.793812 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:30.814746 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:16:30.837457 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:30.898927 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:30.952235 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:30.954013 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:16:30.955675 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:30.959398 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:30.961833 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:30.965989 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:30.967581 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:30.987951 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:31.117041 31032 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:31.117233 31032 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:31.117879 31032 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:16:31.119645 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:31.123425 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:16:31.123540 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:31.123806 31032 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:16:31.166898 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:16:31.169306 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:16:31.202651 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:31.245528 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:31.247015 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:31.267136 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:31.269666 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:31.285713 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:31.289175 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:31.339968 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:31.443635 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:31.448508 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:31.477255 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:31.486369 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:31.493217 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:31.559473 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:31.587213 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:31.604127 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:31.696940 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:31.749493 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:31.760684 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:31.762511 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:31.771836 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:31.775300 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:31.778034 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:31.780637 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:31.795224 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:31.810530 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:31.831385 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:32.073875 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:32.076361 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:32.098299 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:32.104352 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:32.113682 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:32.135417 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:32.172451 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:32.208999 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:32.210443 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:32.264272 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:32.275457 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:32.278993 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:32.319005 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:32.319556 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:32.348232 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:32.354995 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(30b4f075057e42d0bb3f8cdabdfbc643): perf score=0.033341
I20240501 08:16:32.364121 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:32.447795 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(30b4f075057e42d0bb3f8cdabdfbc643) complete. Timing: real 0.089s user 0.049s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":983,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":799,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1454,"threads_started":2,"wal-append.queue_time_us":1276}
I20240501 08:16:32.449122 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(30b4f075057e42d0bb3f8cdabdfbc643): 4922 bytes on disk
I20240501 08:16:32.449751 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(30b4f075057e42d0bb3f8cdabdfbc643) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20240501 08:16:32.482054 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:16:32.495620 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:32.556041 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:32.571564 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:32.589030 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:32.589628 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:32.602995 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:32.608060 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:32.612234 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:32.615873 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:32.621229 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:32.622040 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:32.665508 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:32.697603 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:16:32.716293 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:32.911269 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:32.911522 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:32.912269 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:32.924341 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:32.941826 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:32.953850 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:32.957253 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:33.017091 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:16:33.081485 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:33.087973 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:33.119906 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:33.129416 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:33.134195 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:33.152758 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:33.172641 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:33.298624 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:16:33.398003 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:33.458503 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:33.459033 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:33.477525 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:33.480928 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:33.496515 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:33.554437 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:33.554787 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:33.630923 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:33.682365 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:33.711017 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:33.717397 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:33.735507 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:33.737443 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:33.741314 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:33.747212 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:33.872198 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:33.888603 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:33.964007 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:33.968255 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:33.998695 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:34.022490 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:34.023032 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:34.035459 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:34.063602 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:34.067353 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:34.072737 30582 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:34.072961 30582 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:34.073904 30582 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:16:34.079649 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:34.119473 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:34.133038 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:16:34.136246 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:16:34.140276 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:34.140486 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:34.140813 30582 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:16:34.200392 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:16:34.221055 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:34.246183 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:34.276971 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:34.287724 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:34.298179 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:34.299432 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:34.306900 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:34.312862 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:34.341192 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:34.455448 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:34.495994 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:34.512300 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:34.513314 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:34.513612 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:34.515705 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:34.527860 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:34.535003 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:34.566582 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:34.834048 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:34.847060 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:34.860020 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:34.883946 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:34.901427 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:34.915139 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:34.915980 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:34.943642 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:34.948999 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:34.997813 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:16:35.017561 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:35.028585 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:35.060680 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:35.067384 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:35.085942 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:35.087007 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:35.131836 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:16:35.162575 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:35.167771 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:16:35.176954 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:16:35.201087 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:16:35.307058 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:35.318519 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:35.345372 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:35.351969 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:35.355921 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:35.379839 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:16:35.384617 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:35.419669 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:16:35.420603 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:35.567713 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:35.591565 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:35.631507 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:35.644359 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:35.660837 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:35.668066 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:35.699595 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:35.730633 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:35.798502 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:35.835247 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:35.838392 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:35.856153 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:35.861897 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:35.865684 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:16:35.867933 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:35.870632 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:35.871671 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:35.957988 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240501 08:16:35.958974 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:35.975793 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:16:36.079674 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:16:36.128131 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:36.134833 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:36.147377 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:36.175011 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:36.193540 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:36.227216 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:36.265198 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:36.295241 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:36.313288 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:36.360845 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:36.370019 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:36.374837 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:36.374889 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:36.399307 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:36.454016 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:36.462632 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:36.567931 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:36.654909 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:36.656324 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:36.704593 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:36.705972 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:36.723850 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:36.749162 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:36.756384 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:36.766227 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:36.819481 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:36.834515 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:36.844965 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:36.921123 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:36.975303 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:36.995304 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:37.026142 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:37.060520 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:37.065238 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:37.069484 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:37.104737 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:37.106129 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:37.223150 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:37.240108 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:37.251067 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:37.251443 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:37.304729 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:37.305364 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:37.317778 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:37.366454 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:37.507790 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:37.528512 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:37.531553 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:37.531626 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:37.532656 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:37.542436 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:37.549260 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:37.551308 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:37.561872 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:37.566586 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:16:37.576987 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:37.667862 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:16:37.696769 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:37.726640 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:37.744200 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:37.747452 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:37.759564 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:37.775434 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:37.801668 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:37.806035 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:37.869803 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:37.929529 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:16:37.972311 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:37.974586 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:37.976457 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:38.001933 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:38.010435 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:38.024912 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:38.088519 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:38.092128 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:16:38.261678 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:38.277091 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:38.278127 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:38.288666 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:38.301735 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:38.327826 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:16:38.339210 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:38.357383 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:38.358510 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:38.361598 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:38.388634 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:38.409276 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:38.467768 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:38.537405 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:38.564083 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:38.682552 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:38.682965 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:38.710036 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:38.731082 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:38.733750 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:38.764983 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:38.835661 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:38.932296 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:38.942649 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:38.962677 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:16:38.975443 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:38.976675 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:38.984957 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:39.003293 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:16:39.022208 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:39.023582 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:39.126619 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:39.141988 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:39.192828 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:39.196419 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:39.201498 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:39.209498 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:39.222230 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:39.231637 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:39.268121 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:39.304795 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:39.407487 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:39.455472 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:16:39.510437 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:39.529466 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:39.530838 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:39.547873 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:39.549185 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:39.572470 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:16:39.596138 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:39.610776 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:39.660953 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:39.670244 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:39.799129 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:39.801193 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:39.810132 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:39.829283 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:39.830343 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:39.851509 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:39.865556 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:39.949998 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:39.978600 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:39.997114 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:40.079914 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:40.139039 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:40.151813 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:40.165112 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:40.169977 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:40.174539 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:40.181986 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:40.184192 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:40.204066 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:40.209498 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:40.213167 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:40.222888 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:40.241562 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:16:40.272080 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:40.312222 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:40.313390 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:40.328195 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:40.362563 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:40.365506 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:16:40.405583 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:16:40.425823 32371 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:40.426023 32371 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:40.426786 32371 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:16:40.427839 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:16:40.429911 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:16:40.430191 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:40.434429 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:40.434625 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:40.434986 32371 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:16:40.525352 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:40.525346 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:40.538739 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:40.540836 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:40.546753 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:40.551942 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:40.770701 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:40.789110 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:40.854643 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:40.869199 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:40.887976 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:40.888880 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:40.913564 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:40.919030 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:40.937805 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:40.945983 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:40.949754 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 08:16:40.964143 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:40.966668 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:40.978459 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:40.980605 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:40.986546 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:16:41.166224 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:16:41.172729 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:41.268594 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:41.279589 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:41.280002 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:41.281800 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:41.282541 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:41.301142 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:41.344321 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:41.367736 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:41.368520 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:41.411813 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:41.414615 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:41.539961 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:41.540834 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:41.586565 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:41.587947 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:41.588474 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:41.614245 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:41.725957 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:41.735597 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:41.745525 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:41.775347 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:41.779404 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:41.788087 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:41.801596 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:41.805490 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:41.805635 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:41.811271 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:41.818931 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:41.859661 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:41.877959 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:42.042500 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:42.052577 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:42.070936 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:42.078006 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:42.099862 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:42.110519 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:42.240046 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:42.249123 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:42.281071 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:42.288441 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:42.296365 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:42.306903 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:42.310570 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:42.329483 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:42.351205 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:42.386565 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:42.467587 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:42.510664 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:42.568066 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:42.570366 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:42.571123 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:42.571640 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:42.594596 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:42.595599 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:42.601248 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:42.608677 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:42.634596 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:16:42.637872 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:42.647840 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:42.677161 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:42.877061 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:42.883747 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:42.895269 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:42.918884 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:42.925446 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:42.941730 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:42.954023 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:16:42.969450 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:42.975143 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:43.024701 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:43.055619 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:43.068204 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:43.080289 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:43.100284 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:43.128350 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:43.130661 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:43.228685 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:43.289729 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:43.290124 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:43.304832 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:43.306418 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:43.310706 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:43.317162 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:43.525882 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:43.550395 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:43.558115 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:43.569252 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:43.569684 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:43.582495 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:43.738205 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:43.746675 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:43.750113 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:43.790043 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:43.796296 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:43.810570 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:43.811646 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:43.824709 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:43.941120 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:43.946060 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:43.971540 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:44.006461 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:44.010005 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:44.015201 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:44.024366 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:44.040809 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:44.043085 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:44.063984 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:44.064029 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:44.104074 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:44.180682 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:44.254105 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:44.320822 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:44.321238 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:44.330650 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:44.350569 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:44.351456 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:44.363391 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:44.364389 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:44.364542 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:44.379343 32371 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:44.379562 32371 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:44.380297 32371 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:16:44.400694 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:44.439956 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:44.444437 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:44.473112 8892 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.25.107.7:33929, user_credentials={real_user=slave}} blocked reactor thread for 70798.7us
W20240501 08:16:44.481241 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:44.491639 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:44.496675 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:16:44.496857 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:44.498795 32371 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:16:44.513681 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:44.516887 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:44.531947 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:44.686942 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:44.735927 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:44.737752 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:44.751945 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:44.759373 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:44.768648 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:44.774001 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:44.872251 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:44.941524 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:44.966149 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:44.996546 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:45.009250 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:45.011953 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:45.031283 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:45.034997 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:45.063467 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:45.069465 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:45.088546 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:45.092904 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:45.094642 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:45.174867 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:45.175827 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:45.272557 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:45.272971 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:45.276786 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:45.288996 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:45.302861 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:45.314559 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:45.323870 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:16:45.351095 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:45.437237 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:16:45.477344 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:45.534443 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:16:45.542862 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:45.546162 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:45.553220 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:45.617265 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:45.637903 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:45.647166 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:45.659701 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:45.729279 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:45.748869 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:45.759742 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:45.781921 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:45.792773 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:45.818701 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:45.824483 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:46.028255 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:16:46.032255 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240501 08:16:46.062605 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:46.067046 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:46.077383 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:46.108641 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:46.129302 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:46.161839 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:46.185909 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:46.197017 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:46.296896 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:46.329473 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:46.333554 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:46.356139 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:46.369069 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:46.380429 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:46.383087 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:46.442421 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240501 08:16:46.444275 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:46.474187 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:46.489270 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:46.518525 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:46.525820 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:46.533494 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:46.549198 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:46.551307 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:46.561411 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:46.562084 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:46.564922 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:46.621765 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:46.658005 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:46.690821 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:46.828004 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:46.832813 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:46.848078 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:46.860805 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:46.887136 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:46.894593 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:46.900822 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:46.930922 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:47.008394 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:47.032562 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:47.083607 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:47.100598 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:47.101723 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:47.108682 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:47.114987 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:47.229876 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:47.347623 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:47.368850 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:47.392503 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:47.399942 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:47.417919 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:47.441884 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:47.459085 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:47.519325 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:47.553491 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:47.567577 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:47.626446 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:47.627974 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:47.630322 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:47.665766 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:16:47.668725 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:47.669597 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:47.670184 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:47.684116 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:47.685478 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:47.689883 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:47.772683 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:47.822439 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:47.862602 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:47.882376 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:47.895475 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:47.902628 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:47.931689 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:47.935911 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:47.978873 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:47.980039 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:48.107309 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:48.175748 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:48.209029 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:48.233063 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:48.235432 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:48.238435 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:48.240700 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:48.241657 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:48.251960 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:16:48.467204 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:48.485451 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:48.505203 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:48.508287 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:48.519443 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:48.520774 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:48.670531 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:48.673581 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:16:48.742905 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:48.742947 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:48.757864 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:48.765668 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:48.783514 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:16:48.785663 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:48.805624 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:48.810099 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:48.893158 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:48.896170 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:48.910748 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:48.933925 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:48.940444 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:48.955993 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:48.991109 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:48.996404 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:48.998309 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:49.008862 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:49.178957 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:16:49.210887 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:49.211787 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:49.222992 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:49.248354 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:49.253747 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:49.254128 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:49.278167 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:49.454864 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:49.455253 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:49.511363 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:49.537281 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:49.539494 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:49.549867 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:49.571494 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:49.590365 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:49.626094 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:49.675913 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:49.741124 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:49.743785 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:49.750020 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:49.760457 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:49.770694 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:49.772594 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:49.773555 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:49.781013 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:49.791769 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:49.939795 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:49.958896 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:49.995119 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:50.035359 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:50.068873 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:50.086310 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:50.125813 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:50.134996 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:50.152575 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:50.172979 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:50.245558 1285 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:16:50.245769 1285 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:16:50.246100 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:50.246506 1285 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:16:50.247419 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:50.250619 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:16:50.257601 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:50.267391 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:50.267583 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:16:50.267885 1285 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:16:50.273460 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:50.278048 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:50.278751 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:50.296936 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:50.321226 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:50.415562 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:50.419217 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:50.458331 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:50.493477 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:50.506114 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:50.511713 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:50.514277 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:50.514969 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:50.531723 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:16:50.532809 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:16:50.548144 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:50.579241 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:16:50.600581 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:50.791409 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:50.792140 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:50.814602 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:50.816151 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:50.826828 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:50.829142 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:16:50.839758 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:50.843461 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:50.844036 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:50.903903 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:50.949266 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:50.955026 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:50.973520 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:51.098469 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240501 08:16:51.113291 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:51.152755 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:16:51.253054 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:51.264406 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:51.340310 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:51.356869 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:51.357035 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:51.362496 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:51.370636 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:51.378973 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:51.395119 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:51.395457 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:51.397002 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:51.476528 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(57ecbb30f7764fceb008baaa2c37da8c): perf score=0.033350
W20240501 08:16:51.486836 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:51.516089 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:51.542243 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(57ecbb30f7764fceb008baaa2c37da8c) complete. Timing: real 0.065s user 0.054s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1360,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":2225,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":951,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4008,"threads_started":2,"wal-append.queue_time_us":2873}
I20240501 08:16:51.543974 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(57ecbb30f7764fceb008baaa2c37da8c): 4581 bytes on disk
I20240501 08:16:51.544729 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(57ecbb30f7764fceb008baaa2c37da8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240501 08:16:51.569116 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:51.569631 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:51.573726 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:51.578472 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:51.585682 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:51.612210 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:51.662701 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:51.670754 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 08:16:51.732188 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:16:51.846217 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:51.865715 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:51.873370 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:51.901573 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:51.927631 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:51.944497 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:51.945966 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:52.000069 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:52.013646 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:16:52.076779 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:52.089118 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:16:52.091168 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:52.103945 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:52.172349 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:52.172403 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:52.192699 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:52.194595 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:52.326735 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:52.334656 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:52.337774 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:16:52.338629 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:52.344198 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:52.354281 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:52.359323 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:52.410480 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:52.510043 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:52.608827 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:52.616870 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:52.627377 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:52.657276 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:52.659530 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:16:52.660050 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:52.675112 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:52.702605 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:52.726055 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:52.758107 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:52.758296 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:52.762687 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:52.764686 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:52.774011 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:52.794013 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:52.794466 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:52.819363 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:52.858026 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:16:52.910682 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:52.983304 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:53.046299 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:16:53.091333 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:53.107936 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:53.108844 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:53.113551 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:53.126932 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:53.133252 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:53.150624 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:53.270974 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:53.375293 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:53.377768 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:53.396015 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:53.396041 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:53.430639 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:53.437996 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:53.475476 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 66161.3us
W20240501 08:16:53.488785 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:53.492274 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:53.543002 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:53.554831 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(8b7f87b6032d45c5be5d748ab2cc71e2): perf score=0.033401
I20240501 08:16:53.558928 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:53.576279 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:53.615658 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:53.626168 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:53.663409 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:53.677402 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:53.692360 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:53.707183 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(8b7f87b6032d45c5be5d748ab2cc71e2) complete. Timing: real 0.151s user 0.071s sys 0.044s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":66053,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":920,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1032,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":72400,"threads_started":2,"wal-append.queue_time_us":6350}
I20240501 08:16:53.708806 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(8b7f87b6032d45c5be5d748ab2cc71e2): 4921 bytes on disk
I20240501 08:16:53.710628 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(8b7f87b6032d45c5be5d748ab2cc71e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
W20240501 08:16:53.839442 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:53.878038 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:53.891486 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:53.913595 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:53.921870 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:53.924597 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:53.935904 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:53.940158 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:54.003716 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:54.038015 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:54.140583 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:54.146634 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:54.149679 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:54.156781 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:54.159410 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:54.161986 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:54.167831 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:54.171815 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:54.183465 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:54.331300 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:54.381862 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:54.407110 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:54.439474 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:54.470618 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:54.487162 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:54.493264 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:54.504370 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:54.514199 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:54.700126 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:54.704108 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:54.764060 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:54.764124 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:54.769740 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:54.797012 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:54.814528 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:54.822369 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:54.944116 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:54.944401 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:54.953209 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:54.956386 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:54.964702 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:54.971210 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:54.983883 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:55.069412 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:55.102244 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:55.140280 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:55.166184 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:16:55.206823 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:55.227012 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:55.243141 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:16:55.250052 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:55.262895 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:55.265906 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:55.271396 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:55.272179 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:55.346962 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:55.401718 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:55.411192 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:55.415409 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:55.432404 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:55.433317 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:55.446064 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:55.452363 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:55.457758 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:55.480345 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:55.533861 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:16:55.534955 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:55.580096 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:16:55.662060 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:55.676695 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:55.678186 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:55.696709 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:55.717692 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:55.720882 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:55.724648 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:55.948156 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:55.963563 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:56.003870 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:56.008198 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:56.009083 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:56.014998 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:56.023625 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:56.143303 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:16:56.144466 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:56.154848 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:56.164676 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:16:56.166523 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:56.180003 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:56.202180 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:56.208729 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:56.229419 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:56.247027 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:56.248370 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:56.283830 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:56.336977 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:56.436529 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:56.451985 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:56.453197 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:56.460721 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:56.484444 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:56.485087 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:56.488858 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:56.669423 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:56.746377 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:56.781577 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:56.785063 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:56.785771 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:56.789407 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:56.806571 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:56.820333 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:56.830526 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:56.838601 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:56.846963 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:56.868081 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:56.899946 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:16:56.902694 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:56.909013 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:56.934079 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:56.940135 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:56.944164 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:56.962061 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:56.981721 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:16:57.199107 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:57.248227 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:57.323556 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:57.341382 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:57.370059 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling FlushMRSOp(049090443eaa4b18ba90cb60af9a87ab): perf score=0.033393
I20240501 08:16:57.411939 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:57.413676 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:57.426520 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:57.432411 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:57.506367 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:57.539602 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: FlushMRSOp(049090443eaa4b18ba90cb60af9a87ab) complete. Timing: real 0.169s user 0.072s sys 0.051s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8137,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":857,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":921,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":87029,"threads_started":2,"wal-append.queue_time_us":79102}
I20240501 08:16:57.541010 8741 maintenance_manager.cc:419] P aba7c226aaca4e8c90931f547d48aaa3: Scheduling UndoDeltaBlockGCOp(049090443eaa4b18ba90cb60af9a87ab): 4581 bytes on disk
I20240501 08:16:57.549618 8632 maintenance_manager.cc:643] P aba7c226aaca4e8c90931f547d48aaa3: UndoDeltaBlockGCOp(049090443eaa4b18ba90cb60af9a87ab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
W20240501 08:16:57.601783 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:57.637195 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:57.642633 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:57.649760 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:57.665911 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:57.666508 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:57.687480 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:57.696003 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:57.710206 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:57.747493 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:57.753623 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:57.767027 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:57.787719 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:57.930307 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:57.944224 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:57.945864 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:57.953447 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:57.954540 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:57.961329 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:16:58.020083 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:58.037371 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:58.072724 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:16:58.126122 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:58.137300 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:58.179229 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:58.182607 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:58.194885 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:58.208909 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:58.251109 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:58.300845 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:16:58.348575 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:58.433349 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:58.438598 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:58.446134 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:58.451256 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:58.457227 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:58.484158 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:16:58.491003 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:58.510300 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:16:58.537251 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:16:58.627740 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:58.727195 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:16:58.731637 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:58.757910 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:58.763011 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:58.766754 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:58.782331 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:58.787730 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:58.808151 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:58.823858 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:58.881160 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:58.895987 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:58.954578 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:58.969089 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:16:58.973495 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:58.975553 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:16:59.138490 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:16:59.171100 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:59.178025 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:59.196380 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:59.200874 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:59.225349 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:59.250205 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:59.252535 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:59.274075 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:16:59.306128 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:16:59.418192 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:59.488050 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:59.502486 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:59.502969 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:59.506429 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:59.537482 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:59.567339 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:16:59.586532 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:16:59.646903 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:59.688133 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:59.690140 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:16:59.704139 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:16:59.714583 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:16:59.721501 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:16:59.726313 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:16:59.743664 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:16:59.752923 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:16:59.782348 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:16:59.868577 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling FlushMRSOp(30b4f075057e42d0bb3f8cdabdfbc643): perf score=0.033349
I20240501 08:16:59.912721 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: FlushMRSOp(30b4f075057e42d0bb3f8cdabdfbc643) complete. Timing: real 0.044s user 0.023s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":373,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":942,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":824,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":12672,"thread_start_us":578,"threads_started":2,"wal-append.queue_time_us":424}
I20240501 08:16:59.914191 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(30b4f075057e42d0bb3f8cdabdfbc643): 4922 bytes on disk
I20240501 08:16:59.915066 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(30b4f075057e42d0bb3f8cdabdfbc643) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240501 08:16:59.963057 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:16:59.964015 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:16:59.980994 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:00.001016 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:00.004077 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:00.019703 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:00.029428 32371 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:00.029625 32371 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:17:00.030647 32371 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:17:00.043151 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:00.047878 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 08:17:00.072749 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:17:00.078756 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:00.078941 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:00.079366 32371 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:17:00.090307 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:17:00.154714 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:17:00.196638 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:17:00.232486 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:00.235769 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:00.241030 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:00.242408 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:00.251338 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:00.261345 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:00.335557 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:00.356475 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:17:00.402993 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:17:00.445422 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:00.490981 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:00.491838 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:00.501447 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:17:00.510581 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:00.512338 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:00.515712 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:00.541307 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:00.542483 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:17:00.543569 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:17:00.634600 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:17:00.758971 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:00.771538 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:00.777681 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:00.779848 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:00.797730 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:17:00.825145 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:00.846623 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:00.852528 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:00.893944 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:00.915277 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:00.954725 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:00.958176 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:00.958954 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:00.962015 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:01.008165 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:17:01.043941 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:17:01.045696 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:01.172930 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:01.177372 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:01.201076 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:01.205092 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:01.215299 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:01.217866 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:17:01.221139 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:01.271996 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:17:01.311072 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:17:01.375838 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:01.436935 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:01.467303 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:01.485324 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:01.486501 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:01.497279 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:17:01.523303 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:01.529220 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:01.547308 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:01.549043 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:01.574285 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:01.624256 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:01.657398 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:01.663455 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:01.672637 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:01.684697 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:01.747445 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:01.904103 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:01.915782 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:01.920562 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:01.929028 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:01.940460 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:17:01.941355 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:01.941638 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:01.975065 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:01.997151 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:02.002928 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling FlushMRSOp(658f2a172a2d47c8bf8b44c9d51132e4): perf score=0.033349
W20240501 08:17:02.023065 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:17:02.039661 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:17:02.069365 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:02.098770 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: FlushMRSOp(658f2a172a2d47c8bf8b44c9d51132e4) complete. Timing: real 0.095s user 0.055s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":534,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":8576,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":828,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":20304,"threads_started":2,"wal-append.queue_time_us":20039}
I20240501 08:17:02.100286 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(658f2a172a2d47c8bf8b44c9d51132e4): 4921 bytes on disk
I20240501 08:17:02.101531 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(658f2a172a2d47c8bf8b44c9d51132e4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
W20240501 08:17:02.134588 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:17:02.190670 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:02.268987 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:02.273605 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:02.277026 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:02.315992 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:02.317466 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:02.361423 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:02.410490 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:02.415377 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:02.415524 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:02.428771 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:02.435213 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:02.450821 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:02.468595 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:02.537967 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:17:02.593691 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:17:02.601354 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:02.678468 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:02.700269 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:02.700255 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:02.722853 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:02.737695 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:02.737748 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:02.752198 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:17:02.917498 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:02.942723 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:02.952726 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:02.954427 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:02.958138 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:02.958886 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:02.963301 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:02.963338 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:02.975673 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:02.978977 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:02.984323 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:03.079881 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling FlushMRSOp(049090443eaa4b18ba90cb60af9a87ab): perf score=0.033386
I20240501 08:17:03.118975 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: FlushMRSOp(049090443eaa4b18ba90cb60af9a87ab) complete. Timing: real 0.039s user 0.032s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":594,"dirs.run_cpu_time_us":304,"dirs.run_wall_time_us":1539,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":855,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":729,"threads_started":2,"wal-append.queue_time_us":366}
I20240501 08:17:03.119967 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(049090443eaa4b18ba90cb60af9a87ab): 4581 bytes on disk
I20240501 08:17:03.120558 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(049090443eaa4b18ba90cb60af9a87ab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240501 08:17:03.221686 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:03.221871 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:03.240360 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:03.242444 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:03.243166 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:03.272349 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:03.289484 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:17:03.357863 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:17:03.455731 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:03.495458 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:17:03.500962 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:03.524699 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:03.536051 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:03.540017 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:03.545776 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:03.567591 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:03.615239 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:03.647606 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:03.662724 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:03.668291 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:03.710196 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:03.724030 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:03.728935 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:03.742413 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:03.756599 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:03.759830 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:03.835425 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:03.872366 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:04.014966 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:04.015815 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:04.023775 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:04.031719 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:04.042755 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:04.063529 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:04.095844 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:04.180985 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:17:04.210870 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:17:04.268874 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:04.308995 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:04.319455 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:04.328238 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:04.347911 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:04.353410 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:04.363191 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:04.363899 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:04.428421 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:04.515092 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:04.521728 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:04.548168 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:04.554471 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:04.564851 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:04.577503 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:04.601079 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:04.648674 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:04.667804 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:17:04.692756 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:04.709220 3632 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:04.709429 3632 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:17:04.710161 3632 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:17:04.712924 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:17:04.724886 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:04.725068 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:04.725315 3632 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:17:04.866838 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:04.872114 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:04.894542 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:04.966395 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:04.978232 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:04.997756 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:05.003228 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:05.017669 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:05.022241 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:05.031687 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:05.036084 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:05.070451 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:05.071398 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:05.091281 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:05.124845 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:05.293649 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:05.348093 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:05.354138 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:05.356096 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:05.361888 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:05.369824 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:05.374779 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:05.375128 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:05.379720 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:05.381721 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:05.385706 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:05.472770 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:05.643108 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:05.646533 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:17:05.669968 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:05.690371 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:05.708834 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:05.721675 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:05.742452 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:05.766340 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:05.769435 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:17:05.779207 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:05.783874 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:05.802600 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:05.806701 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:05.880398 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:05.890219 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:05.891180 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:05.907487 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:05.920029 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:06.101984 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:06.121237 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:06.132855 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:06.142305 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:06.293152 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:17:06.303783 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:06.335884 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:17:06.340137 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:06.366693 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:06.375272 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:06.386391 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:06.400171 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:06.410216 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:06.421700 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:06.464754 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:06.468691 3632 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:06.468878 3632 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:17:06.469516 3632 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:17:06.478425 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:17:06.489785 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:06.499128 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:06.517498 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:06.517781 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:06.518079 3632 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:17:06.520128 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:06.527732 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:06.528910 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:06.685771 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:06.690801 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:06.691154 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:06.744597 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:06.756927 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:06.768270 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:06.782835 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:06.789548 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:06.799405 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:06.815608 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:06.831593 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:06.957585 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:06.967579 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:17:07.020085 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:07.020110 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:07.041389 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:07.048882 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:07.060158 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:07.075620 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:07.194389 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:07.242096 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:07.261197 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:07.265410 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:07.266685 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:07.271569 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:07.280148 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:07.301374 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:07.326581 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:07.351920 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 75540.2us
W20240501 08:17:07.375376 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:07.410612 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:07.533360 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:07.565861 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:07.577664 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:07.587155 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:07.594664 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:07.620057 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:07.634392 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:07.660467 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:07.711977 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:07.736035 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:07.744252 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:07.749914 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:07.769377 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:07.778337 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:07.782665 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:07.790246 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:07.812976 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:07.845506 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:07.845788 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:08.007534 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:08.025548 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:08.038524 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:08.046504 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:08.053987 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:08.064572 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:08.070384 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:08.084395 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:08.293845 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:17:08.302433 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:08.332434 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:08.365996 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:08.369948 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:08.387537 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:08.399613 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:17:08.428984 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:08.434245 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:08.446584 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:08.447731 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:08.465698 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:08.491649 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:08.518849 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:08.532919 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:08.552722 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:08.558485 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:08.587998 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:08.625447 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:08.626456 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:08.682824 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:08.793318 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:08.802389 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:08.808801 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:08.819380 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:08.822875 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:08.823246 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:08.842655 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:08.867902 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:09.057545 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:09.059135 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:09.089917 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:09.091032 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:09.103261 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:09.109154 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:09.132968 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:09.137166 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:09.154997 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:09.350060 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:09.354825 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:09.358191 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:09.371613 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:09.399237 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:09.403196 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:09.414342 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:09.490650 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:09.553941 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:09.601395 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:09.622903 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:09.646312 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:09.659919 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:09.670629 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:09.673336 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:09.674367 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:09.684945 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:09.700870 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:09.725250 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:09.838599 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:09.843636 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:09.858726 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:09.875061 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:09.880590 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:09.895252 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:09.980602 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:10.000159 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:10.130841 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:10.150539 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:10.163851 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:10.165994 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:10.187628 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:10.213886 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:10.220817 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:10.223512 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:10.240324 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:10.345081 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:10.350183 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:10.361256 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:10.384202 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:10.408824 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:10.417416 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:17:10.435431 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:10.528689 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:10.564960 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:10.567957 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:10.584519 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:10.593168 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:10.593863 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:10.619922 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:10.621759 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:10.648886 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:17:10.657045 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:10.778980 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:10.851485 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:10.958113 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:10.977874 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:10.995050 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:11.008644 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:11.010020 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:11.038074 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:11.044467 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:11.094835 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:11.099148 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:11.120982 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:11.137403 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:17:11.171885 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:11.198658 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:11.218037 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:11.221414 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:11.222487 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:11.272636 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:11.275915 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:11.277561 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:11.282590 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:11.405362 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:17:11.406406 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:11.427109 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240501 08:17:11.505803 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:11.515650 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:11.530854 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:11.534742 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:11.547336 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:11.566123 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:11.570940 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:11.628304 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:11.735692 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:11.741551 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:11.766467 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:11.794656 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:11.794924 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:11.804037 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:11.830114 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:11.843681 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:11.877645 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:11.989243 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:17:11.996208 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:12.014700 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:12.025540 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:12.035717 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:12.047601 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:12.064692 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:12.066812 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:12.067898 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:12.109400 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:12.238965 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:12.290772 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:12.303357 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:12.304390 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:12.307862 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:12.328282 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:12.329309 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:12.409000 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:12.414158 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:12.414207 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:12.444412 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:12.534152 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:12.539990 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:12.546950 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:12.564496 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:12.584465 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:12.592638 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:12.594472 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:12.598362 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:17:12.745661 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:12.829648 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:12.839941 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:12.854141 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:12.864024 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:12.864794 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:12.890069 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:12.904685 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:12.969991 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:13.024636 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:13.029552 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:13.031159 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:13.042520 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:13.052196 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:13.067164 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:13.084668 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:13.265304 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:13.295372 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:13.314414 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:13.314618 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:13.321066 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:13.342288 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:13.351348 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:13.372678 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:17:13.493948 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:13.495505 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:13.503170 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:13.517108 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:13.547499 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:17:13.555812 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:13.562933 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:13.563544 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:13.584839 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:13.585215 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:13.641571 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:13.681886 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:13.754546 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:13.764535 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:13.792374 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:13.793097 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:13.795127 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:13.836072 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:13.853256 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:13.873598 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:13.912943 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:13.966177 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:14.104696 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:14.109620 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:14.133199 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:14.138556 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:14.143590 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:14.168190 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:14.185421 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:14.198585 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:14.264003 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:14.286944 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:14.300130 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:14.306607 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:14.312109 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:14.325651 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:14.334717 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:14.339795 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:14.531289 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:14.598464 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:14.607656 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:14.613332 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:14.632120 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:14.634752 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:14.636598 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:14.651558 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:14.678490 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:14.694553 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:14.804756 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:14.815826 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:14.833029 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:14.835608 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:14.879779 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:14.907485 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:14.908963 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:14.980186 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:14.994980 5059 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:14.995186 5059 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:17:14.995802 5059 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:17:14.998190 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:14.998754 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:17:14.998849 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:14.999068 5059 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:17:15.041393 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:15.065146 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:15.088716 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:15.090214 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:15.123288 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:15.126108 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:15.129657 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:15.134246 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:15.144757 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:15.194289 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:17:15.204527 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:15.346977 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:15.354746 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:15.355250 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:15.369239 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:15.388080 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:15.396798 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:15.413249 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:15.502245 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:15.587438 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:15.597353 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:15.615243 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:15.619477 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:15.626886 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:15.637447 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:15.705926 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:17:15.706950 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:15.740132 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:15.751861 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:17:15.762041 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:15.802209 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:17:15.911945 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:15.912988 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:15.922700 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:15.934540 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:15.958449 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:15.967881 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:16.005646 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:16.030611 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:16.037956 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:16.078438 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:16.096261 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:16.102041 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:16.155580 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:16.155629 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:16.205094 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:17:16.295195 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:16.350687 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:17:16.354769 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:16.372287 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:16.374070 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:16.393900 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:16.396782 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:16.410248 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:16.411420 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:16.424472 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:17:16.431427 5367 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:16.431670 5367 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:17:16.432695 5367 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:17:16.440099 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:17:16.450935 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:17:16.477396 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:16.480351 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:17:16.481981 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:17:16.482685 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:16.483115 5367 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:17:16.539310 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:16.584656 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:16.688982 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:16.704399 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:16.710593 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:16.712587 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:16.723201 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:16.762935 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:16.832396 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:16.873582 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:16.894101 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:16.901506 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:16.918430 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:16.925360 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:16.937717 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:16.941799 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:16.942600 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:16.965488 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:17.086546 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:17:17.100925 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:17:17.136336 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:17.194742 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:17.204399 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:17.208760 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:17.214766 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:17.227545 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:17.250975 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:17.251150 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:17.376231 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:17.426687 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:17.434689 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:17.435185 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:17.475522 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:17.478762 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:17.480011 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:17.525524 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:17.550736 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:17.631829 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:17.641340 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:17.649258 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:17.667155 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:17.688835 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:17.705363 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:17.721042 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:17.727260 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:17.747875 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:17.775877 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:17:17.854452 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:17.877290 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:17.963232 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:17.969633 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:17.989379 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:18.003188 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:18.007236 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:18.008486 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:18.092603 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:18.179468 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:18.181419 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:18.199929 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:18.210532 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:18.212865 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:18.223429 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:18.281996 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:18.426452 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:18.487637 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:17:18.502868 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:18.524706 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:18.558521 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:18.559551 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:18.561944 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:17:18.567402 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:18.572666 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:18.657524 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:18.708995 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:18.722895 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:18.787838 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:18.791810 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:18.802682 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:17:18.812896 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:18.818645 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:18.856292 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:18.864912 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:18.868902 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:18.885769 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:18.904693 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:18.971799 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:17:18.979758 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:19.000195 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:19.001538 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:19.006444 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:19.010504 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:19.025077 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:19.203581 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:19.265311 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:19.282313 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:19.296406 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:19.301160 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:19.301342 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:19.312522 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:19.324100 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:19.337771 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:19.340374 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:19.470360 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:19.540735 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:19.567202 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:19.570989 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:19.571130 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:19.633731 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:19.637817 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:19.655357 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:19.676062 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:19.724328 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:19.784305 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:19.790761 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:19.799695 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:19.822440 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:19.824206 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:17:19.827203 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:19.898833 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:19.992630 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:20.020644 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:20.093274 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:20.114523 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:20.149909 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:20.157083 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:20.178969 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:20.199115 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:20.202790 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:20.206136 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:20.212040 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:20.239634 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:20.240378 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:20.246233 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:20.323858 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:20.338842 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:20.364878 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:20.389060 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:20.393245 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:20.393610 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:20.558329 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:20.580596 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:20.581701 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:20.611518 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:20.620994 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:20.652118 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:20.653040 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:20.744634 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:20.746668 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:17:20.783833 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:20.886209 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:20.896109 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:20.916535 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:20.928992 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:20.949821 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:20.972443 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:20.989583 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:21.003655 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:21.038628 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:21.043278 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:21.076331 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:21.080231 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:21.098627 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:17:21.122568 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:21.143958 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:21.177515 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:21.184707 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:21.202916 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:21.297613 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:21.384325 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:21.434648 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:17:21.457140 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 08:17:21.522787 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:21.549441 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:21.569998 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:21.593775 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:21.631413 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:21.636587 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:21.638120 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:21.663213 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:21.667378 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:21.677685 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:21.690006 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:21.696549 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:17:21.698496 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:21.790690 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:21.807291 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:21.832294 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:21.865906 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:22.000444 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:22.017903 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:22.018855 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:17:22.052060 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:22.082655 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:22.089550 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:22.089841 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:22.102397 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:22.141373 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:22.155472 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:17:22.198242 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:22.248739 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:22.308899 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:22.315150 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:22.323978 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:22.345431 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:22.366056 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:22.367502 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:22.376305 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:22.436230 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:22.532018 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:22.550463 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:22.556895 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:22.562850 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:22.586539 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:22.604022 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:22.608228 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:22.619724 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:22.744824 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:22.749861 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:22.827598 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:22.845135 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:17:22.846721 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:22.876493 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:22.887646 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:22.892963 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:22.903525 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:22.924922 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:22.954219 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:22.972669 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:22.998625 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:23.020546 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:23.026635 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:23.081552 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:23.188678 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:23.225831 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:23.262951 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:23.273552 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:23.277735 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:23.288347 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:23.299437 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:23.300112 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:23.331991 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:23.445160 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:17:23.523836 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:23.573930 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:23.597496 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:23.598030 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:23.627053 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:23.627142 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:23.646756 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:23.678644 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:23.682054 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:23.687253 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:17:23.695029 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:23.700452 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:23.700608 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:23.717824 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:23.720789 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:23.726843 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:23.751930 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:23.752274 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:23.771775 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:23.772207 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:23.826306 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:23.886693 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:24.006963 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:24.016346 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:24.019368 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:24.034379 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:24.052593 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:24.103009 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:24.142365 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:24.241137 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:24.243319 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:24.270607 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:24.283711 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:24.295935 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:24.299141 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:24.302538 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:24.307049 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:24.452128 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:24.486400 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:24.526655 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:24.542904 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:24.557528 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:24.561450 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:24.564519 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:24.568359 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:24.582850 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:24.586381 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:24.649329 6372 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:24.649533 6372 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:17:24.650221 6372 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:17:24.651669 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:24.656541 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:17:24.656706 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:24.657270 6372 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:17:24.657761 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:24.775933 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:24.777197 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:24.914811 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:24.918682 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:24.936460 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:24.950927 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:24.964038 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:24.976110 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:24.981551 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:25.007375 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:25.009444 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:25.041646 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:25.081064 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:25.086321 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:25.104231 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:25.104733 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:25.208597 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:25.245952 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:25.264245 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:25.327188 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:25.330730 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:25.331477 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:25.341660 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:25.347366 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:25.369199 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:25.370121 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:25.608451 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:25.624805 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:25.651044 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:25.678159 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:25.692296 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:25.699992 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:25.748422 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:17:25.751008 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:17:25.778571 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:25.791610 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:25.807641 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:25.821070 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:17:25.823273 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:25.830073 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:25.831758 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:25.838771 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:25.913386 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:25.923925 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:25.924136 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:17:25.959542 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:17:26.100239 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:26.131786 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:26.138070 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:26.139916 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:26.179664 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:26.185446 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:26.187615 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:26.211817 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:26.221809 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:26.233399 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:17:26.273770 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:26.312885 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:26.385541 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:26.403575 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:26.404389 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:26.423552 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:26.435200 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:26.436697 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:17:26.441589 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:26.453167 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:26.473362 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240501 08:17:26.570909 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:26.586557 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:26.594491 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:26.607529 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:26.611549 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:26.634610 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:26.664348 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:26.669595 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:26.742244 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:26.804178 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:26.880486 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:26.894997 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:26.910530 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:26.945318 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:26.956099 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:26.967175 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:26.970062 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:27.052654 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:27.131682 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:27.149022 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:27.149920 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:27.164881 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:17:27.165930 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:27.206892 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:27.211967 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:27.227964 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:27.256951 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:27.269068 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:27.315564 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:17:27.360371 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:27.424207 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:27.430006 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:27.441332 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:27.447520 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:27.447906 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:27.452818 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:27.478389 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:27.487202 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:27.517024 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:27.736057 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:27.757119 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:27.763388 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:27.787572 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:27.788388 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:27.795389 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:27.815666 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:17:27.841332 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:27.920713 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:27.925016 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:27.944861 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:27.950695 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:27.957937 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:27.966614 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:27.967064 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:27.989813 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:28.117234 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:28.133328 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:28.238250 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:28.256008 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:28.269883 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:28.313899 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:28.320753 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:28.339627 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:28.417243 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:28.418900 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:28.432722 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:28.455615 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:28.456573 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:28.487020 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:28.537103 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:28.552735 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:28.554090 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:28.562111 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:17:28.598237 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:17:28.626835 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:17:28.668069 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:28.708189 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:28.739895 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:28.757611 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:28.770051 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:28.774693 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:28.795943 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:28.960714 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:28.982054 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:29.007167 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:29.023574 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:29.029754 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:17:29.041611 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:29.041741 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:29.047475 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:29.063654 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:29.080282 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:29.168920 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:17:29.187728 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:29.228009 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:29.230108 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:29.242758 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:29.252405 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:29.267644 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:29.286203 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:29.342624 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:29.489207 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:29.510715 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:29.513742 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:29.514500 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:29.519140 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:29.536899 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:29.543941 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:29.629263 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:29.636755 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:17:29.777168 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:29.792361 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:29.803233 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:29.823788 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:29.841706 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:29.842519 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:29.884398 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:29.885044 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:29.908155 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:29.942572 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:29.964519 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:29.997474 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:30.009176 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:30.024372 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:30.038405 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:30.046384 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:30.072017 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:30.088559 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:30.095716 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:30.175387 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:17:30.269454 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:30.372546 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:30.381188 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:30.386499 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:30.393836 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:30.398989 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:30.407738 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:30.438745 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:30.542106 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:30.630089 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:30.635555 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:30.641847 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:30.642987 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:30.659495 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:30.660825 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:30.674831 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:17:30.728312 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:17:30.814543 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:17:30.933574 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:30.940439 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:30.963464 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:30.972569 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:30.986662 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:31.015071 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:31.019572 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:31.019867 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:31.030341 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:31.068289 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:31.112963 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:31.117312 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:31.140525 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:17:31.173180 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:17:31.179419 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:31.230615 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:31.250846 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:31.252300 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:31.265275 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:31.428862 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:31.437520 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:31.449343 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:31.465013 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:31.465873 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:31.469301 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:31.480129 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:31.512985 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:31.555578 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:17:31.575676 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:17:31.589293 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:17:31.603055 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:31.745628 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:31.750304 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:31.757261 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:31.774627 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:31.805943 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:31.820030 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:31.856135 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:31.863813 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:31.949695 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:31.972028 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:31.975777 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:31.987172 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:31.999788 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:32.003118 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:32.029985 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:32.043704 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:32.070453 7532 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:32.070638 7532 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:17:32.071468 7532 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:17:32.085954 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:17:32.092298 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:17:32.092681 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:32.092877 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:32.093166 7532 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:17:32.116909 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:32.158551 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:17:32.168565 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:17:32.172644 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240501 08:17:32.299507 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:32.322338 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:32.325021 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:32.339107 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:32.356761 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:32.363564 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:32.374398 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:32.374537 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:32.476289 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:32.486804 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:32.523377 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:32.528607 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:32.544397 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:32.551712 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:32.552959 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:32.579777 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:32.630679 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:17:32.681113 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:32.685228 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:17:32.753314 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:17:32.797816 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:32.801674 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:32.808526 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:32.824151 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:32.833561 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:32.835768 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:32.870596 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:33.146353 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:33.156906 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:33.158319 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:33.174510 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:33.191532 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:33.194314 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:33.198793 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:33.207481 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:17:33.213971 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:33.223845 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:33.242458 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:33.251369 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:33.260960 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:33.297272 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:33.319517 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:33.464666 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:17:33.512447 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:17:33.682510 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:33.748562 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:17:33.769290 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:17:33.802664 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:33.813373 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:33.821974 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:33.849078 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:33.877595 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:33.888407 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:33.902554 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:33.915176 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:33.916349 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:33.916543 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:33.969739 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:33.981343 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:34.011693 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:34.030047 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:34.082391 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:34.159412 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:34.177096 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:17:34.193608 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:34.229612 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:34.253383 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:34.256587 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:34.283082 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:34.288699 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:34.303189 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:34.303444 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:17:34.409544 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:34.526528 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:34.560096 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:34.570526 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:34.572048 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:34.595072 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:34.596060 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:34.613503 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:34.642724 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:34.658514 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:17:34.685850 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:34.734987 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:34.749258 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:34.751137 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:34.753345 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:34.760977 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:34.772692 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:34.772856 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:34.796754 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:34.874392 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:34.893788 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:17:34.932139 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:35.019187 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:35.030637 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:35.044919 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:35.101402 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:35.135466 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:35.137429 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:35.151484 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:35.255975 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:35.258854 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:35.339856 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:35.345726 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:35.354521 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:35.359916 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:35.362478 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:35.379050 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:35.397068 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:35.402628 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:35.534454 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:35.537636 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:35.553838 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:35.570228 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:35.603868 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:35.613257 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:35.640444 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:35.667671 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:35.832105 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:35.832537 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:35.846021 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:35.852500 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 67 similar messages]
I20240501 08:17:35.853308 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:35.853893 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:35.931509 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:35.935737 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:36.006362 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:36.035727 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:36.044035 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:36.069753 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:36.074004 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:36.088610 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:36.146826 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:36.146824 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:36.151921 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:36.174448 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:17:36.288904 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:36.295092 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:36.306159 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:36.315573 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:36.331404 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:36.341894 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:36.390488 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:36.405205 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:36.516006 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:36.582715 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:17:36.591431 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:36.594381 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240501 08:17:36.594857 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:36.600144 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:36.604882 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:36.611259 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:36.613126 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:36.741231 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:36.756179 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:36.773468 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:36.777714 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:36.784907 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:36.840655 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:36.852120 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:36.857203 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:36.866515 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:36.879650 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:36.932298 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:36.933912 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:36.954093 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:37.031885 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:37.145408 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:37.175171 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:37.188369 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:37.202952 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:37.208314 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:17:37.216275 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:37.225005 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:37.230970 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:37.237831 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:37.242796 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:37.274510 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:37.276715 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:17:37.297883 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:37.329793 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:37.386109 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:37.444304 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:37.496383 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:37.552163 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:17:37.579979 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:37.588341 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:37.590893 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:37.591380 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:37.597328 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:37.616679 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:37.654691 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:37.666469 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:37.769203 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:37.808925 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:37.870110 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:37.888211 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:37.915656 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:37.920639 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:37.938563 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:37.959985 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:37.998525 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:38.007140 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:38.036686 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:38.046641 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:17:38.150947 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:38.165453 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:38.175873 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:38.183653 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:38.190559 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:38.198875 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:38.260738 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:38.369966 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:38.464354 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:38.478240 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:38.489650 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:38.512018 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:38.513088 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:38.531448 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:38.533248 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:17:38.610757 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:38.622321 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:38.635843 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:38.662101 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:38.666836 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:17:38.677767 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:38.680004 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:38.688486 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:38.734723 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:17:38.735200 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:38.911993 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:38.919872 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:38.923836 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:38.925611 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:38.932966 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:38.959295 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:38.960011 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:39.015849 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:39.076643 9339 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:39.076853 9339 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:17:39.077591 9339 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:17:39.088856 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:17:39.090301 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:17:39.096292 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:39.096449 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:39.096762 9339 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:17:39.167162 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:39.186959 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:17:39.216768 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:17:39.219172 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:39.260329 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:39.260733 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:39.276754 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:39.278193 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:39.304554 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:39.307843 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:39.338310 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:39.411078 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:39.411800 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:39.415170 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:39.420526 6833 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 44796.4us
I20240501 08:17:39.452033 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:39.452622 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:39.476928 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:39.478981 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:39.488061 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:17:39.587680 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:39.623183 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:39.732395 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:39.914839 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:39.924469 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:39.955235 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:39.970434 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:39.973839 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:17:39.979640 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:39.982201 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:40.040145 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:40.050741 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:40.077930 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:40.083617 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:40.101078 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:17:40.102320 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:40.120746 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:17:40.131340 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:17:40.162708 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:40.177361 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:40.181409 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:40.218065 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:17:40.369886 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:40.377638 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:40.381686 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:40.392637 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:40.414367 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:40.420157 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:40.484982 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:40.499137 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:40.509982 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:40.567581 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:40.704771 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:40.713872 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:40.727555 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:40.738292 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:40.741678 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:40.781014 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:40.800938 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:40.842396 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:40.890909 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:40.894711 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:40.920284 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:40.951804 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:40.951807 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:40.960335 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:40.963409 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:40.969105 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:40.984161 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:41.001382 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:17:41.035254 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:17:41.225404 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:41.279064 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:41.303032 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:41.317163 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:41.321738 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:41.333047 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:41.339931 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:41.375056 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:41.381879 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:41.384160 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:41.389437 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:41.427434 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:41.430027 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:41.460628 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:41.483065 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:41.590121 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:17:41.620337 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:17:41.700335 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:41.704402 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:41.708909 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:41.719533 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:41.736515 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 08:17:41.746603 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:41.768252 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:41.776182 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:41.791108 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:41.835429 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:41.937355 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:17:41.985090 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:41.988296 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:42.014036 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:42.017884 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:42.033229 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:42.039911 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:42.082685 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:17:42.092541 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:42.109962 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:42.179693 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:42.259001 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:42.263257 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:42.279243 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:42.282666 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:42.294241 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:42.304608 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:42.359675 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:17:42.373699 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:42.429569 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:17:42.448701 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:42.467180 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:42.497087 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:42.506855 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:42.511545 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:42.511562 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:42.543830 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:42.544871 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:42.574039 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:42.578675 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:42.743201 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:42.774806 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:42.779198 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:42.779888 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:42.802376 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:42.806658 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:42.811304 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:42.853969 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:42.864997 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:42.974825 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:43.012085 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:43.014381 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:43.017818 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:43.019043 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:43.019995 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:43.052907 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:43.065250 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:43.274099 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:43.281770 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:17:43.313555 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:43.322909 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:43.342550 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:43.353579 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:43.370064 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:43.372124 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:43.389645 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:43.392093 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:43.418326 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:43.419648 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:43.482281 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:43.488857 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 52442us
I20240501 08:17:43.499126 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:43.560333 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:43.625968 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:17:43.637015 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:17:43.739188 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:43.748845 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:43.749951 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:43.759371 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:43.761090 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:43.764539 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:43.765460 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:43.766327 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:43.767678 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:43.778299 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:43.923895 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:44.011101 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:44.057089 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:44.068701 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:44.102696 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:44.114428 10429 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:44.114624 10429 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
W20240501 08:17:44.116806 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:17:44.117094 10429 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:17:44.143383 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:44.143940 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:44.144297 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:44.148854 10325 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:17:44.139703 6704 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 101324us
I20240501 08:17:44.184784 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:44.185233 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:44.214653 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:44.232146 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:44.235787 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:44.236490 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:44.257316 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:44.263607 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:44.279839 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:44.287596 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:44.329176 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:44.332829 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:44.391203 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:44.513149 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:44.513581 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:44.523578 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:44.523981 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:44.527757 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:44.544540 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:44.549069 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:44.571082 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:44.759567 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:44.771698 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:44.808579 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:44.815866 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:44.825901 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:44.826531 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:44.830726 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:44.860901 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:44.867172 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:44.917663 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:44.936349 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:44.959923 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:44.963620 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:44.997048 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:45.000473 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:45.039408 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:45.054916 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:45.054937 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:45.126525 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:45.225318 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:45.249737 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:45.253010 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:45.260852 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:45.270852 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:45.273943 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:45.286999 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:45.287307 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:45.416517 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:17:45.439118 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:45.458926 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:45.467900 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:45.490566 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:45.512984 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:45.563313 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:45.613754 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:45.654049 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:45.679556 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:45.685688 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:45.736564 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:45.747572 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:45.753752 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:45.766021 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:45.770025 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:45.790076 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:45.914137 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:45.936244 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:46.010763 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:17:46.012030 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:46.113057 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:46.128329 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:46.130477 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:46.138609 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:46.139350 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:46.139928 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:46.143764 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:46.144354 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:46.144672 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:46.178119 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:46.192699 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:46.202546 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:46.216156 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:46.230010 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:46.280916 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:46.332841 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:17:46.401201 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:46.498220 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:46.504770 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:46.508771 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:46.510025 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:46.518344 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:46.521272 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:46.530679 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:46.648715 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 08:17:46.686100 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:46.736722 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:46.746524 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:46.749644 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:17:46.814918 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:46.823413 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:46.832649 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:46.834043 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:46.859071 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:46.903086 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:46.935256 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:46.972157 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:46.981586 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:46.983426 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:46.994347 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:47.045758 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:47.052119 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:47.079268 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:47.079560 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:47.155376 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:47.157223 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:47.266446 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:47.279971 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:47.288614 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:47.295656 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:47.307979 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:47.334904 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:47.349468 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:47.380462 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:47.512035 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:47.551692 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:47.555158 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:47.562810 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:47.583585 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:47.590190 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:47.597016 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:17:47.666034 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:47.686460 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:17:47.691303 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:47.707168 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:47.713757 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:47.753542 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:47.768942 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:47.779630 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:47.816540 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:47.818792 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:47.819656 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:47.847350 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:47.874522 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:47.879720 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:48.094080 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:17:48.115658 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:48.123214 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:48.125236 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:48.137091 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:48.143682 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:48.156998 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:48.173542 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:48.189270 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:48.193248 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:48.321909 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:48.334798 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:48.336405 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:48.348837 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:48.375174 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:48.376372 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:48.387634 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:48.553988 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:48.606371 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:48.618546 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:48.625180 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:48.662757 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:48.667892 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:48.673417 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:48.686772 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:48.687178 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:48.710855 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:48.722196 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:48.960130 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:48.976828 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:48.984493 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:49.008417 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:49.016273 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:49.020737 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:49.039036 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:49.080459 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:49.080519 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:17:49.120383 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:49.120975 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:17:49.121380 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:49.125392 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:49.129109 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:49.151125 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:49.206142 6833 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 58970us
W20240501 08:17:49.221381 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:49.331866 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:49.419380 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:49.426178 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:49.432415 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:49.443888 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:49.452358 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:49.467253 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:49.468611 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:49.473341 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:49.490372 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:49.629192 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:49.658360 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:17:49.718176 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:49.740036 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:49.740015 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:49.745028 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:17:49.792829 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:49.796224 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:49.810971 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:49.831758 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:49.877871 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:49.903400 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:49.903934 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:49.949827 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:49.949882 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:49.964921 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:49.983963 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:50.008989 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:50.129467 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:17:50.183252 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:50.188097 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:50.206199 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:50.207746 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:50.215097 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:50.229363 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:17:50.230417 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:50.352018 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:50.352516 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:50.380462 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:50.408162 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:50.412817 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:50.414496 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:50.417747 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:50.455488 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:50.459182 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:50.461748 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:50.469880 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:17:50.514964 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:50.546936 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:50.628836 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:50.629616 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:50.635471 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:50.639022 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:50.647481 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:50.655671 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:50.668879 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:50.811131 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:17:50.901945 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:50.925161 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:50.946273 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:50.962496 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:50.966580 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:50.974809 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:51.021814 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:51.025974 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:51.131240 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:51.134737 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:51.145449 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:51.145570 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:51.155373 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:51.163975 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:51.165120 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:51.183373 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:51.240635 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:51.280514 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:51.368274 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:17:51.435974 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:51.451766 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:51.455525 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:51.462790 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:51.475792 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:51.482476 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:51.627099 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:51.661669 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:51.675321 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:17:51.729868 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:51.753101 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:51.773340 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:17:51.786922 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:51.790551 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:51.808620 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:17:51.810742 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:51.840008 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:51.864877 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:17:51.892064 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:51.920014 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:51.938519 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:51.967140 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:51.967876 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:51.969471 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:17:51.984588 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:51.992049 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:52.079574 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:52.121846 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:52.142570 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:52.162019 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:52.186607 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:17:52.219781 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:52.233634 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:52.237475 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:52.253430 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:52.281057 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:52.284087 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:52.285861 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:52.420305 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:52.422804 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:52.440225 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:52.465943 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:52.479669 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:52.547260 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:52.668773 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:17:52.719508 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:52.720077 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:52.720657 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:52.737015 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:52.737457 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:52.758447 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:52.778981 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:52.793663 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:52.814384 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:52.838348 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:52.842916 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:52.899820 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:52.930904 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:52.960317 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:17:52.963764 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:52.981173 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:52.992718 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:53.051327 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:53.096323 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:53.097365 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:53.097582 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:53.141175 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:17:53.189649 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:53.190173 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:53.210502 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:17:53.213655 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:53.259852 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:53.282567 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:53.295316 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:53.343977 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:53.463804 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:53.471689 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:53.474951 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:53.485806 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:53.493463 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:53.526546 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:53.542606 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:53.605616 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:53.699259 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:17:53.788883 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:53.806170 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:17:53.818632 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:53.833456 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:53.843824 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:53.844748 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:53.858362 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:53.864343 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:53.873638 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:53.891523 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:53.905902 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:17:53.920536 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:53.930579 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:53.938959 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:53.952203 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:54.290985 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:54.342437 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:54.343242 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:54.354336 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:54.379004 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:54.425294 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:54.473173 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:17:54.523049 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:54.528589 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:54.530992 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:54.560806 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:54.561622 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:54.576506 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:54.578784 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:54.590564 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:54.591732 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:54.643808 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:17:54.647663 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:54.669126 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:54.673306 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:54.707345 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:54.800446 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:54.817725 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:54.850986 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:54.894565 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:54.902865 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:54.904126 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:54.919227 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:54.924379 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:54.952144 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:55.201166 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:55.232705 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:55.237603 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:55.262084 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:55.265358 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:55.300050 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:55.357728 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:55.361717 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:55.386804 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:55.393908 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:17:55.405491 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:55.410624 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:55.416311 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:55.416337 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:55.441991 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:55.443200 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:55.449733 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:55.471027 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:55.525002 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:55.525734 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:55.578390 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:55.641566 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:55.664448 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:55.721539 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:55.726851 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:55.745468 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:55.750559 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:55.752934 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:55.779628 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:55.784076 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:55.799484 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:55.807364 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:55.942770 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:55.958036 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:55.996260 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:56.022951 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:56.024466 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:56.026429 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:56.105726 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:56.112592 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:17:56.229293 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:56.243455 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:56.246996 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:56.247565 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:56.263880 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:56.291275 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:56.306502 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:56.323860 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:17:56.335141 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:56.548586 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:56.563506 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:56.564729 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:56.580953 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:56.581231 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:56.614270 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:56.637594 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:56.643630 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:56.660593 12326 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:56.661011 12326 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:17:56.666412 12326 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:17:56.723459 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:56.724931 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240501 08:17:56.731428 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:17:56.742892 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:17:56.746695 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:56.756902 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:56.760725 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:17:56.760933 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:56.761343 12328 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:17:56.767948 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:56.775544 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:17:56.778014 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:56.835944 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:56.844260 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:56.848933 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:56.871600 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:56.907577 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:56.971793 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:56.982215 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:56.982730 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:56.988170 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:56.997749 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:57.008598 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:57.048257 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:57.095672 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:57.176750 12058 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:17:57.176977 12058 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:17:57.177807 12058 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:17:57.179329 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:57.180511 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:17:57.180652 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:17:57.180948 12058 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:17:57.221030 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:57.234472 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:57.246778 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:57.258419 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:57.268718 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:17:57.276789 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:57.285630 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:57.291770 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:57.465338 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:57.481163 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:57.509706 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:57.516955 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:57.538424 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:57.559616 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:57.575898 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:57.576124 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:57.656319 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:57.698225 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:57.757983 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:57.778333 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:57.784557 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:17:57.785806 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:17:57.811240 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:57.817937 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:57.832262 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:57.834441 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:57.835024 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:57.851572 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:57.973415 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:57.979316 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:17:58.003095 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:17:58.003556 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:58.024492 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:58.033046 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:58.045212 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:17:58.049027 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:58.075323 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:58.123451 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:58.170949 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:58.203982 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:58.292503 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:58.303193 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:58.345077 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:58.348325 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:58.363952 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:58.374189 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:58.375375 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:58.399278 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:58.422667 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:17:58.423100 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:58.429661 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:58.435113 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:58.493044 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:58.494644 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:58.508008 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:58.518654 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:17:58.710533 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:58.715636 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:58.735718 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:58.736522 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:58.738519 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:58.754999 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:58.816851 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:58.860899 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:17:58.874286 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:17:58.987825 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:59.003890 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:59.016860 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:59.034795 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:59.047801 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:59.096369 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:59.098407 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:59.120438 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:59.139628 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:59.170482 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:59.172923 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:59.212930 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:59.289227 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:17:59.291219 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:17:59.317909 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:17:59.318676 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:17:59.329205 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:17:59.408038 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:59.416471 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:59.421036 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:59.429454 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:59.431694 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:59.431705 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:17:59.560765 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:59.576937 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:17:59.621234 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:17:59.679246 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:59.685539 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:17:59.700152 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:17:59.707093 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:59.738512 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:59.742230 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:59.755333 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:59.909421 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:17:59.927841 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:17:59.938699 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:17:59.947703 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:17:59.956115 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:17:59.978215 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:00.042539 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:18:00.044791 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:18:00.254948 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:18:00.256217 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:18:00.268469 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:00.269158 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:18:00.269227 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:00.269383 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:00.277573 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:00.284567 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:00.286289 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:00.287022 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:00.291201 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:00.313884 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:00.437019 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:18:00.450525 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:00.501740 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:00.506862 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:00.513379 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:00.520164 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:00.523458 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:00.551774 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:00.554786 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:00.573535 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:00.757247 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:18:00.763355 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:00.777346 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:00.794306 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:00.801625 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:00.819638 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:00.854054 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:00.858275 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:00.862358 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:00.871811 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:00.886612 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:18:00.988809 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:00.997709 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:01.008664 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:01.026098 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:01.028631 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:01.030571 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:01.084756 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:01.194301 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:18:01.232095 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:18:01.232302 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:01.306226 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:01.328903 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:01.343856 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:01.396906 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:01.439147 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:01.462077 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:01.486757 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:01.488914 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:01.514091 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:01.541987 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:01.545218 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:01.562424 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:01.589277 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:01.636174 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:01.663089 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:18:01.689026 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:18:01.791664 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:01.812088 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:01.815443 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:01.822374 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:01.836769 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240501 08:18:01.841877 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:01.872709 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:01.875468 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:01.878290 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:01.894030 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:01.903494 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:18:01.921568 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:18:01.979143 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:02.184916 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:02.197831 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:02.213168 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:02.216603 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:02.281140 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:02.282194 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:02.334415 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:02.353735 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:02.381512 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:02.383090 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:02.399327 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:02.406167 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:02.433251 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:02.482014 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:18:02.537513 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:02.572983 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:02.596274 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:18:02.602903 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:02.616037 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:02.622359 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:02.669278 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:02.673005 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:02.687788 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:02.703934 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:02.863070 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:18:02.864567 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:02.865538 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:02.890350 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:02.939666 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:02.948877 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:02.967249 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:02.968858 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:02.974366 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:02.975932 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:02.996946 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:03.015686 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:18:03.019951 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:03.020006 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:03.047382 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:03.074503 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:03.095525 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:03.114499 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:03.134932 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:03.151244 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:03.189426 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:03.189937 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:18:03.339552 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:03.341837 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:03.488459 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:18:03.573659 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:03.625169 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:03.632858 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:03.640025 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:03.642572 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:03.643507 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:03.678980 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:03.708297 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:03.711292 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:03.736770 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:03.754423 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:03.768999 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:03.774322 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:03.780232 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:03.865459 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:18:03.966177 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:04.024425 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:04.027297 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:04.028285 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:04.043419 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:04.066025 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:04.081211 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:04.083199 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:04.204684 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:04.305454 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:04.307950 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:04.313439 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:04.313915 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:04.315845 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:04.328251 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:04.349263 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:04.367266 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:04.375010 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:04.401963 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:04.493888 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:04.501534 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:04.524130 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:04.526875 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:04.530714 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:04.542183 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:04.553122 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:04.762702 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:04.815553 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:04.817821 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:04.832870 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:04.834683 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:04.912559 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:04.913642 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:04.914280 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:04.916908 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:05.001251 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:05.029632 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:05.046092 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:05.047886 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:05.053587 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:05.066861 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:05.142944 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:05.263227 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:05.270684 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:05.278005 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:05.278184 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:05.285400 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:05.306766 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:05.319875 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:05.330538 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:05.331471 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:05.345976 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:05.367595 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:05.386837 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:05.393472 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:05.410557 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:05.465883 13314 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:05.466115 13314 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:05.466866 13314 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:05.474874 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:18:05.487143 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:18:05.490990 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:05.491149 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:05.491437 13314 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:18:05.549289 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:05.597183 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:05.608302 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:05.613543 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:05.636564 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:05.644666 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:05.647847 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:05.649283 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:05.661247 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:05.670214 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:05.675491 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:05.687036 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:05.697340 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:05.718420 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:05.737260 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:05.738166 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:05.762527 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:05.824378 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:05.960969 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:18:05.999621 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:06.006393 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:06.026355 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:06.041113 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:06.054847 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:06.088603 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:06.100823 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:06.239812 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:18:06.267724 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:06.272679 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:06.289562 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:06.292157 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:06.294950 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:06.334409 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:06.431608 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:18:06.466300 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:06.522445 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:06.523514 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:06.523695 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:06.539381 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:06.553321 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:06.555655 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:06.638722 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:18:06.682833 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:06.690515 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:06.792865 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:06.816409 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:06.824258 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:06.843839 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:06.862656 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:06.866739 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:06.867882 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:06.892634 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:06.903103 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:06.926373 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:06.934857 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:06.935889 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:06.954582 6833 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 57136.1us
W20240501 08:18:06.969022 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:18:06.974812 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:06.975004 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:07.031778 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240501 08:18:07.038811 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:07.114924 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:07.130905 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:07.136310 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:07.322849 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:07.326874 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:07.330410 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:07.348248 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:07.359050 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:07.373775 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:07.378969 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:07.434839 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:07.615694 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:07.628425 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:07.634902 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:07.645228 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:07.650522 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:07.659063 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:07.660326 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:07.700992 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:07.701952 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:07.711803 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:07.759958 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:07.760054 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:07.762473 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:07.770807 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:07.780685 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:07.792680 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:07.807633 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:07.808036 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:07.886683 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:18:07.887567 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:18:07.970695 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:07.999914 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:18:08.017895 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:08.031482 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:08.034972 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:08.046881 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:08.062206 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:08.089762 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:08.092200 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:08.097084 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:08.120071 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:08.136817 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:08.215601 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:18:08.258476 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:08.334581 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:08.345299 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:08.353454 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:08.385063 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:18:08.391752 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:08.419485 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:08.420673 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:08.435582 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:08.481559 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 57583.7us
I20240501 08:18:08.605077 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:08.605700 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:08.605729 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:08.609110 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:08.615561 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:08.648690 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:08.782495 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:08.852670 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:08.873960 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:08.897207 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:08.920195 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:08.925658 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:08.926106 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:08.962584 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:09.002187 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:18:09.022176 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:09.127925 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:09.128587 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:09.138396 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:09.139645 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:09.143038 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:09.171952 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:09.208051 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:18:09.260215 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:18:09.273465 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:18:09.290611 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:18:09.336439 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:09.354485 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:09.359674 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:09.376451 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:09.377497 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:09.382879 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:09.386905 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:09.392933 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:09.633522 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:09.640642 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:09.647856 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:09.705127 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:09.708443 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:09.711318 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:09.730849 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:09.736937 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:09.766742 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:09.855345 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:09.887543 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:09.887548 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:09.888343 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:09.892904 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:09.900950 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:09.913220 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:09.953804 14145 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:09.953985 14145 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:09.954635 14145 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:09.956044 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:18:09.957806 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:09.957937 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:09.958290 14145 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:18:10.019860 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:18:10.137938 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:18:10.139636 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:18:10.140290 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:10.220547 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:10.232065 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:10.237268 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:10.237862 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:10.242317 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:10.246995 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:10.265195 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:10.269692 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:10.296036 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:10.319272 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:10.335512 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:10.339447 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:10.348420 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:10.355531 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:10.379346 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:18:10.379546 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:18:10.543485 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:18:10.563819 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:10.598714 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:10.614709 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:10.622963 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:10.644107 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:10.659029 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:10.662547 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:10.662585 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:10.663621 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:18:10.667001 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:10.687992 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:10.751580 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:10.921878 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:10.953575 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:10.957502 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:10.971573 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:10.997202 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:11.009214 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:11.035890 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:11.103698 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:11.108609 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:11.134758 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:11.160415 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:11.173779 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:11.175873 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:11.189858 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:11.198743 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:18:11.282191 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:18:11.362622 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:18:11.362782 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:11.380443 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:11.391439 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:11.395053 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:11.399644 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:11.406543 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:11.448446 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:18:11.532704 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:18:11.688333 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:11.709906 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:11.730384 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:11.730526 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:11.738744 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:11.759743 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:11.766327 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:11.797180 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:11.800864 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:11.801463 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:11.834692 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:11.850644 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:11.875627 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:11.894364 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:11.904986 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:11.923729 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:11.924227 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:12.025414 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:18:12.112574 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240501 08:18:12.169063 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:12.288065 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:12.302451 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:12.308604 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:12.309724 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:12.310038 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:12.327824 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:12.359468 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:12.372548 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:12.390241 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:12.482795 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:12.539186 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:12.540957 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:12.547147 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:12.548220 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:12.550487 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:12.562150 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:12.564663 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:12.570247 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:12.606961 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:18:12.621891 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:12.805316 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:12.834362 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:12.862617 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:12.872429 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:12.888999 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:12.903841 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:12.907106 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:12.937762 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:18:12.945883 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:12.954978 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:12.976835 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:13.000329 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:13.017233 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:13.030920 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:13.063083 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:13.070008 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:13.079488 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:13.106273 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:18:13.149716 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:18:13.205272 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:13.280232 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:13.295972 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:13.296101 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:13.306339 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:13.310415 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:13.311158 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:13.315136 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:13.328920 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:18:13.347606 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:13.435564 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:13.561509 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:13.573324 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:13.614771 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:13.638625 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:13.638913 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:13.664435 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:13.671240 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:13.707800 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:13.759459 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:13.765305 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:13.772356 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:13.776760 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:13.794550 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:13.798650 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:13.817070 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:13.901896 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:18:13.904335 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:18:14.023736 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:14.025619 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:14.041554 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:14.062736 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:14.070560 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:14.083175 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:14.141211 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:14.221711 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:18:14.245636 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:14.272708 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:14.299356 14819 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:14.299527 14819 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:14.300213 14819 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:14.305794 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:14.307057 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:18:14.307206 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:14.307646 14819 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:18:14.330425 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:14.342638 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:14.344612 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:14.361204 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:14.367082 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:14.371742 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:14.418152 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:14.476612 14707 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:14.476861 14707 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:14.477699 14707 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:14.485844 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:18:14.500000 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:18:14.504611 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:14.504869 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:14.505183 14707 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:18:14.522442 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:14.541450 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:14.550603 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:14.566416 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:14.601141 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:14.607420 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:14.607498 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:14.660038 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:14.835500 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:14.866772 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:14.867049 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:14.869498 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:14.870112 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:14.878755 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:14.887611 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:14.912484 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 46985.4us
I20240501 08:18:14.916129 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:15.045094 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:15.047988 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:15.069144 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:15.076517 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:15.077580 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:15.098371 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:15.103230 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:15.104568 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:15.128996 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:15.139019 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:18:15.153786 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:15.289816 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:15.337389 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:15.357513 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:15.363420 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:15.376350 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:15.390532 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:15.399139 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:15.452463 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:15.453588 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:15.531221 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:15.633162 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:18:15.636215 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:15.647282 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:15.649262 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:15.669664 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:15.692615 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:15.718645 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:15.723616 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:15.741397 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:15.741672 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:15.754591 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:15.763903 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:15.781795 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:15.844131 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:15.856575 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:15.856899 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:15.859901 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:15.863322 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:15.957876 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:18:15.966179 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:16.096802 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:16.105046 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:16.122026 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:16.124981 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:16.137806 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:16.179126 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:18:16.181479 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:16.218811 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:16.234791 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:16.304992 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:18:16.306701 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:16.401800 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:16.417872 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:16.451792 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:16.465505 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:16.487323 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:16.529042 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:16.553092 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:16.610059 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:16.614085 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:16.636186 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:16.684561 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:16.705286 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:16.717880 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:16.731549 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:16.734318 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:16.777602 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:16.870012 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:16.894085 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:16.927506 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:16.946713 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:16.949816 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:16.952586 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:16.959153 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:16.970629 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:17.041736 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:18:17.169624 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:17.178840 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:17.179471 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:18:17.200232 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:17.208158 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:17.208153 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:17.213095 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:17.232498 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:17.347384 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:17.358283 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:17.396263 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:17.415119 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:17.421962 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:17.431524 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:17.434237 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:17.446377 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:17.452558 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:17.577615 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:17.622789 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:17.646672 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:17.673251 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:17.698141 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:17.712456 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:17.758210 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:17.758937 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:17.811422 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:17.815564 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:17.866868 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:17.877032 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:18:17.904366 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:17.921411 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:17.926875 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:17.927983 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:17.941884 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:17.946076 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:17.951958 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:17.953277 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:18.011420 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:18:18.012599 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:18:18.099653 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:18.210863 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:18.249707 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:18.259482 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:18.261863 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:18.266360 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:18.266575 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:18.296829 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:18.297508 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:18.311703 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:18.313298 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:18:18.370275 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:18.433951 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:18.490942 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:18.524279 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:18.529073 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:18.546411 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:18.548199 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:18.571990 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:18.595772 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:18.705519 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:18.737841 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:18.740907 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:18.754034 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:18.757108 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:18.800446 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:18.800633 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:18.801029 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:18.825837 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:18.861011 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:18:19.140777 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:19.146072 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:19.152072 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:19.168664 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:19.207720 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:19.225272 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:19.230485 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:19.231748 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:19.232822 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:19.276841 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:19.283567 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:19.283614 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:19.318794 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:19.342900 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:19.348992 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:19.384774 14707 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:19.384963 14707 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:19.385666 14707 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:19.388208 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:19.388377 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:18:19.388504 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:19.388756 14707 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:18:19.420993 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:19.536302 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:19.536732 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:19.536913 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:19.541256 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:19.559911 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:19.565248 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:19.617657 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:19.736310 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:19.784837 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:19.828809 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:19.840831 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:19.860677 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:19.870734 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:19.903316 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:19.909451 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:19.918921 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:19.939646 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:19.949419 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:19.954770 14819 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:19.954950 14819 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:19.955608 14819 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:19.961171 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:18:19.962088 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:19.962232 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:19.962826 14819 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:18:19.998525 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:20.004935 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:20.026161 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:20.032544 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.054368 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:20.115937 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:20.140877 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:20.144893 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:20.173199 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:20.305127 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:20.306097 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:20.331312 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:20.332361 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:20.348917 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.364451 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:20.382714 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:20.382740 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:18:20.458788 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:20.537961 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:20.545588 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:20.591826 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:20.592343 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:20.608444 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:20.656095 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:20.673120 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:20.696398 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:20.733459 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:20.782408 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:20.783596 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:20.790997 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:20.792706 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:20.792817 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:20.793952 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:18:20.806043 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:20.824376 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:20.843565 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:20.851639 6834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 47461.5us
W20240501 08:18:20.856382 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:20.896525 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:20.922391 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:21.033108 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:18:21.068212 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:21.140244 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:21.144279 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:21.155639 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:21.158231 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:18:21.162020 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:21.167582 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:21.183408 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:21.260593 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:18:21.270574 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:18:21.354384 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:18:21.372980 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:21.385058 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:21.389619 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:21.389894 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:21.392019 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:21.458704 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:21.521418 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:21.658722 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:21.695583 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:21.700937 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:21.709971 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:21.739231 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:21.751655 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:21.766772 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:21.789441 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:21.796855 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:21.799865 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:21.803828 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:21.824095 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:21.843464 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:21.855389 6571 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.3:43505, user_credentials={real_user=slave}} blocked reactor thread for 54728.5us
W20240501 08:18:21.858166 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:22.021862 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:22.060797 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:22.087975 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:22.143435 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:22.145637 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:22.156705 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:22.163134 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:22.171115 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:22.185711 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:18:22.195158 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:22.262545 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:18:22.340593 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:22.408615 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:22.425822 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:22.426604 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:22.426620 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:22.445899 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:22.467648 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:22.470983 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:22.489357 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:22.494302 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:18:22.509127 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:22.623695 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:22.626767 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:22.649468 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:22.652734 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:22.655961 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:22.667819 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:22.688228 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:22.760064 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:22.920794 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:22.951200 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:22.975768 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:22.979326 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:22.982115 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:23.009330 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:23.015887 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:23.034765 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:23.050282 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:23.053026 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:23.056432 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:23.084287 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:23.098894 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:23.119627 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:23.120272 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:23.174419 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:23.175478 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:18:23.178006 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:23.190455 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:23.282737 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:23.289353 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:23.291561 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:18:23.366317 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:23.404958 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:23.411573 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:23.423987 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:23.425009 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:23.428783 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:23.438206 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:23.464134 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:23.468112 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:18:23.471333 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:18:23.484170 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:23.685243 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:18:23.704602 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:23.705441 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:23.731190 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:23.731411 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:23.775391 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:23.780143 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:23.835685 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:23.855804 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:23.878778 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:23.901070 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:23.920275 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:23.931365 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:23.943250 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:23.996927 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:24.014909 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:24.146435 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:24.317911 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:24.331802 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:24.337424 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:24.373713 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:24.374135 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:24.389636 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:24.504261 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:18:24.534986 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:24.544126 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:24.545800 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:24.546301 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:24.555608 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:18:24.555970 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:24.559473 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:24.575232 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:24.658200 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:24.719781 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:18:24.786365 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:24.788821 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:24.795817 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:24.819545 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:24.824806 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:24.843938 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:24.845054 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:25.001192 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:25.032394 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:25.041296 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:25.046876 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:25.075016 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:18:25.084254 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:25.098541 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:25.113353 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:25.143393 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:25.157011 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:25.201522 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:25.241235 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:25.271832 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:25.276942 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:25.280493 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:25.302062 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:25.310968 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:25.326681 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:25.389144 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:25.438324 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:25.509860 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:18:25.565358 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:25.570454 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:25.624017 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:25.631637 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:25.636437 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:25.670784 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:25.678200 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:25.721050 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:25.726187 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:25.731282 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:18:25.764437 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:25.785379 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:25.786034 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:25.848855 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:25.864142 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:25.909174 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:25.925217 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:25.945942 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:25.947309 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:26.023005 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:26.054306 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:26.057142 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:26.081157 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:26.086867 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:26.088873 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:26.089084 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:18:26.099432 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:26.115265 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:26.165935 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:18:26.316114 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:26.344812 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:26.346518 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:26.347548 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:26.369900 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:26.371076 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:26.399521 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:26.487895 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:18:26.489740 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:26.527231 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:18:26.603302 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:26.613046 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:26.625074 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:26.631191 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:26.642920 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:26.680405 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:26.888705 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:26.889699 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:26.890575 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:26.907753 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:18:26.908146 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:26.922438 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:26.933247 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:26.938548 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:26.969524 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:27.056480 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:27.151407 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:27.156877 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:27.161120 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:27.171301 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:27.173465 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:27.173506 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:27.180871 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:18:27.270576 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:18:27.271467 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:18:27.311547 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:27.313978 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:18:27.320712 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:27.375036 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:27.456849 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:27.480443 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:27.481392 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:27.482978 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:27.502688 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:27.560719 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:27.582072 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:27.661245 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:27.698896 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:27.738981 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:27.740556 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:27.765720 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:27.775391 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:27.796568 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:27.813273 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:27.824327 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:27.830834 6833 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 51118.8us
I20240501 08:18:27.939596 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:27.941895 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:27.957947 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:27.958009 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:27.979269 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:27.983546 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:27.984539 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:28.035686 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:28.084380 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:18:28.089865 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:18:28.223217 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:28.243515 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:18:28.249518 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:18:28.250926 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:28.251915 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:28.273135 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:28.279729 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:28.314075 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:28.326570 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:28.327499 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:18:28.349287 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:18:28.413687 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:28.416206 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:28.422920 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:28.426541 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:28.438169 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:28.443162 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:18:28.465294 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:28.499236 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:28.505015 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:28.533746 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:28.570753 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:18:28.583261 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:28.708549 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:28.709820 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:28.720293 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:28.722528 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:28.733054 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:28.734556 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:28.763695 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:28.993373 6703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 48863.2us
W20240501 08:18:28.995957 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:29.004627 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:29.006578 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:29.030071 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:29.092721 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:29.120256 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:29.121697 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:29.137753 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:29.215399 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:29.236325 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:29.243805 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:29.270783 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:29.290397 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:29.314865 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:29.320506 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:29.368172 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:18:29.558671 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:29.559731 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:29.582129 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:29.607414 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:29.609238 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:29.628288 16586 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:29.628516 16586 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:29.629329 16586 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:29.634397 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:18:29.634852 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:29.634989 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:29.635612 16586 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:18:29.637315 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:29.641160 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:29.678607 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:18:29.802848 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:29.808779 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:29.819005 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:18:29.823776 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:29.832155 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:29.841827 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:29.848378 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:29.863773 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:18:29.863808 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:29.866624 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:18:29.894277 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:18:30.062824 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:30.069980 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:30.088100 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:30.089395 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:30.093642 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:30.114077 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:30.116580 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:30.142215 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:18:30.151619 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:30.240254 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:30.247931 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:30.248051 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:30.335458 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:30.338801 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:30.354482 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:30.368796 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:30.379518 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:30.384383 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:30.603688 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:30.615870 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:30.624433 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:30.632618 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:30.638171 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:30.657402 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:30.662190 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:30.686326 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:30.716693 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:18:30.782456 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:30.811389 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:30.816792 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:18:30.837841 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:18:30.919077 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:18:30.922086 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:30.925235 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:30.937852 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:30.941279 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:30.967730 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:30.983673 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:31.059219 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:31.065667 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:31.075155 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:31.094470 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:31.101430 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:31.118903 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:31.139173 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:31.140192 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:31.265077 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:31.281966 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:31.346273 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:31.346555 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:31.369489 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:31.371353 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:31.381734 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:31.388677 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:18:31.394572 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:31.412191 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:31.430511 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:31.555415 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:18:31.556321 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:31.647688 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:31.662212 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:31.680873 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:31.681672 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:31.708231 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:31.712967 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:31.783403 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:31.851105 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:31.852689 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:31.874734 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:31.891683 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:31.909601 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:18:31.913272 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:31.937142 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:31.988781 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:32.112802 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:18:32.183773 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:32.196141 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:32.220433 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:32.239338 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:32.258832 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:32.262138 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:32.265022 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:32.308856 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:32.312743 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 08:18:32.332713 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:32.345173 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:18:32.352495 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:18:32.389555 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:32.424031 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:32.432605 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:32.439411 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:32.445263 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:32.451258 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:18:32.484117 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:32.525408 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:32.568599 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:32.568568 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:32.704813 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:32.713595 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:32.716820 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:32.735459 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:32.735634 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:32.764731 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:32.765000 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:32.765689 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:33.001595 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:33.006485 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:33.006501 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:33.009339 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:33.034135 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:33.045970 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:33.051002 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:33.189718 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:33.199376 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:33.216068 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:33.226792 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:33.249104 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:33.253557 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:33.270859 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:18:33.272325 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:33.272994 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:33.276957 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:33.299955 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:33.343386 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:33.546689 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:33.550457 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:33.550837 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:33.552486 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:33.581221 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:18:33.589776 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:33.625265 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:33.652832 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:33.664227 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:33.675866 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:33.819644 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:33.828507 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:33.831729 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:33.861606 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:33.863797 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:33.879369 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:33.891115 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:33.898329 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:33.955852 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:18:34.033255 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:18:34.061794 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:34.071138 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:34.179148 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:34.211582 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:34.212193 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:34.259068 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:34.264204 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:34.266309 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:34.276701 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:34.391453 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:34.406471 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:34.413059 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:34.414538 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:34.414515 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:34.417910 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:34.553617 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:34.555027 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:34.563596 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:34.666930 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:34.676873 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:34.679378 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:34.682844 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:34.699827 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:34.714697 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 72764.1us
I20240501 08:18:34.724473 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:34.732008 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:34.771615 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:34.824267 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:34.934998 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:34.952167 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:34.953109 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:34.953119 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:34.981259 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:35.000845 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:35.028764 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:35.051545 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:35.051776 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:18:35.062985 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:18:35.125675 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:35.183416 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:35.194977 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:35.200467 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:35.209931 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:35.217290 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:35.240473 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:35.241073 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:35.269551 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:18:35.434182 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:35.451431 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:35.475832 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:35.476115 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:35.509191 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:35.564146 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:35.564412 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:35.625437 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:35.746870 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:35.753957 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:35.755460 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:35.765595 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:35.770443 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:35.806387 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:35.806952 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:35.809641 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:35.811935 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:35.823325 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:35.954068 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:35.959077 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:35.967280 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:35.981403 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:36.004527 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:36.004685 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:36.094607 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:36.119819 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:36.134480 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:36.299527 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:36.311300 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:36.314554 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:36.331032 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:36.334435 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:36.345105 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:36.348275 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:36.355921 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:18:36.379863 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:36.382856 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:36.402874 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:36.426385 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:36.465579 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:36.470319 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:36.489776 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:36.493539 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:36.527740 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:36.536973 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:36.538834 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:36.685737 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:18:36.797905 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:36.870833 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:37.019270 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:37.035698 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:37.068683 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:37.076838 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:37.084760 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:37.112896 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:37.148514 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:37.151360 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:37.152027 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:37.152071 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:37.170823 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:37.179412 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:37.200017 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:37.230687 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:37.232475 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:37.373387 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:18:37.399101 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:37.408054 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:37.447386 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:18:37.458607 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:37.462064 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:37.466645 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:37.474157 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:37.480875 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:37.497603 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:37.513430 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:37.520174 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:37.563452 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:37.575109 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:37.665658 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:37.685967 17875 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:37.686159 17875 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:37.686929 17875 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:37.688510 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:37.705847 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:18:37.705996 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:37.706458 17875 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:18:37.773810 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:37.813817 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:37.827390 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:37.828717 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:37.845968 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:37.861457 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:37.885985 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:37.919548 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:37.945869 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:37.956956 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:37.962909 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:37.963683 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:37.965864 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:37.971562 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:38.011914 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:38.154243 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:38.220719 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:18:38.248673 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:38.252579 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:38.252719 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:38.269059 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:38.277221 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:38.293473 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:38.298641 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:18:38.300698 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:38.301790 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:38.414309 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:38.489179 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:38.506449 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:38.518504 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:38.548923 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:38.549335 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:38.578711 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:38.581358 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:38.582778 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 57205.6us
I20240501 08:18:38.587531 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:38.590965 17516 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:38.591176 17516 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:38.592192 17516 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:38.608222 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 08:18:38.620855 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:18:38.621730 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:38.621865 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:38.622131 17516 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:18:38.692607 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:38.711367 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:38.730310 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:38.752681 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:38.754823 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:38.755901 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:38.758692 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:38.759487 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:38.846899 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:38.898125 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:18:38.933399 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:38.938827 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:38.945887 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:39.022164 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:39.030413 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:39.036358 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:39.055541 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:39.060678 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:39.103705 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:39.159987 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:39.278404 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:39.280337 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:39.282800 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:39.288437 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:39.307637 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:39.312155 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:39.376739 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:39.478708 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:39.480309 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:39.484256 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:39.494203 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:39.498404 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:39.509235 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:39.520435 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:39.544481 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:39.641476 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:39.737967 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:39.757342 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:39.758540 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:39.775048 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:39.784447 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:39.786625 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:39.792358 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:39.829959 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:39.845073 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:40.010957 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:40.012830 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:40.032977 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:40.034993 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:40.039657 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:40.051353 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:40.058898 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:40.060477 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:40.061443 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:40.126483 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:40.150985 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:40.305114 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:40.332859 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:40.349669 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:40.352281 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:40.376822 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:40.381556 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:40.400240 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:40.434700 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:40.443482 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:40.446892 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:40.461992 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:40.504866 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:40.512630 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:40.528964 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:40.540300 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:40.568212 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:40.727411 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:40.741235 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:40.753286 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:40.762378 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:40.770752 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:40.783340 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:40.783445 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:40.786428 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:40.791565 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:40.811966 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:41.043272 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:41.057211 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:41.082051 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:41.090885 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:41.095885 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:41.117352 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:41.165182 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:41.168550 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:18:41.173671 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:18:41.180469 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:41.182977 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:41.185425 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:41.218056 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:41.227813 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:41.284786 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:41.348219 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:41.355980 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:41.358333 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:41.414880 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:41.499075 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:41.550968 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:41.554788 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:41.561066 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:41.569468 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:41.572598 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:41.621462 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:41.628983 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:41.673178 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:41.701750 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:18:41.915908 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:41.917857 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:41.930318 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:41.950455 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:41.957500 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:41.973359 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:41.975659 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:41.977010 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:42.014678 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:42.027156 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:42.029085 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:42.038420 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:42.066722 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:42.097472 8892 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.3:43505, user_credentials={real_user=slave}} blocked reactor thread for 115288us
I20240501 08:18:42.110870 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:42.145274 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:42.198617 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:42.213577 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:42.225557 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:42.229070 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:42.319919 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:42.331063 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:42.344774 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.356693 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:42.368844 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:42.385322 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240501 08:18:42.386025 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:42.435353 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:42.480934 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:18:42.513509 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:42.518828 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:42.551257 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:42.572546 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:42.585114 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:42.599427 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:42.600368 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.601027 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:42.613394 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:42.624066 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:42.777053 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:42.811841 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:42.812893 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:42.825250 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:42.834600 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:42.845221 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:42.848088 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:42.853308 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:42.868480 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:43.024252 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:43.051362 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:43.152984 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:43.173965 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:43.190167 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:43.207237 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:43.212481 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:43.245046 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:43.254660 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:43.262603 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:43.272884 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:43.290552 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:43.304476 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:43.335103 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:43.337684 6879 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:43.338047 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:43.381542 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:43.385233 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:18:43.403013 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:43.452172 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:43.527915 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:43.619011 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:43.623795 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:43.641844 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:43.650801 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:43.653618 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:43.657063 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:43.659087 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:18:43.774346 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:43.859634 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:18:43.954010 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:43.956861 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:43.963124 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:43.996876 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:43.998871 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:44.018755 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:44.075325 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:44.095872 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:44.115336 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:44.149609 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:44.204147 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:44.205724 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:44.221400 19270 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:44.221608 19270 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:44.222402 19270 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:44.224429 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:44.230717 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:44.242669 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:44.243706 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:18:44.243842 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:44.244094 19270 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:18:44.245074 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:44.321655 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:44.473485 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:44.502521 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:44.510093 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:44.574440 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:44.583356 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:44.592965 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:44.605896 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:44.609230 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:44.628149 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:44.631080 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:44.658515 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:44.734022 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:44.749853 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:44.768533 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:44.804103 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:44.835589 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:44.840684 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:44.843334 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:44.854573 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:44.865962 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:44.954524 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:45.031765 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:45.035918 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:45.051259 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:45.056046 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:45.100633 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:45.102535 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:45.106144 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:45.130215 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:45.196444 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:45.218453 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:45.263797 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:45.303730 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:45.307137 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:45.339413 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:45.346299 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:45.361217 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:45.369465 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:45.439646 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:45.440943 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:45.458393 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:45.543210 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:45.556452 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:45.569851 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:45.581866 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:45.604204 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:45.639113 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:45.657182 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:45.791436 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:45.882041 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:45.886356 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:45.905988 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:45.923261 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:45.978802 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:45.997287 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:45.998539 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:46.081465 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:46.175899 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:18:46.187065 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:18:46.200700 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:46.219477 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:46.240743 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:46.263319 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:18:46.268903 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:46.268944 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:46.308892 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:46.333631 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:46.337564 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:46.387109 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:46.395637 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:46.419440 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:46.421177 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:46.433056 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:46.448390 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:46.448997 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:46.498694 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:46.533442 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:46.656942 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:46.751645 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:46.771456 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:46.774454 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:46.775230 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:18:46.790547 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:46.793874 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:46.826546 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:46.928738 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:46.986196 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:47.022244 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:47.033577 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:47.035034 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:47.046761 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:47.048848 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:47.079109 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:47.128373 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:47.244341 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:47.250392 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:47.252899 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:47.257227 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:47.260159 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:47.275681 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:47.280401 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:47.297642 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:47.369902 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:18:47.399960 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:47.435490 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:18:47.436883 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:47.473886 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:18:47.498594 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:47.506639 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:47.514427 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:18:47.520908 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:47.537746 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:47.549832 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:47.561347 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:47.569532 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:47.570508 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:47.593426 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:47.712114 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:47.728817 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:47.761560 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:47.762379 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:47.772799 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:47.797169 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:47.809484 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:47.891465 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:18:47.893895 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:47.987880 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:47.998482 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:47.998549 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:48.001506 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:48.009052 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:48.017575 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:48.060128 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:48.066649 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:48.309484 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:48.330003 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:48.336246 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:48.347919 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:48.357909 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:48.371908 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:48.379366 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:48.399765 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:18:48.409035 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:48.447728 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:48.451894 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:18:48.480906 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:48.492272 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:48.533500 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:48.563894 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:48.571521 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:48.700290 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:48.792415 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:48.808545 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:18:48.834358 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:18:48.873490 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:48.882601 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:48.894641 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:48.924422 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:48.955552 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:48.961871 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:48.963199 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:48.963471 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:48.971791 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:18:48.973356 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:49.146404 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:49.146947 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:49.155607 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:49.163832 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:49.172164 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:49.174523 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:49.184120 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:49.352131 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:49.412433 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:49.440665 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:49.446583 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:49.449435 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:49.467391 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:49.478771 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:49.491464 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:49.504595 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:49.519934 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:49.562301 19966 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:49.562523 19966 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:49.563222 19966 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:18:49.576429 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:49.580734 19937 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:18:49.580942 19937 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:18:49.581663 19937 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:18:49.587651 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:49.598580 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:18:49.603313 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:49.607450 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:49.620170 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:49.626914 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:18:49.627255 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:49.627966 19966 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:18:49.629760 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:18:49.638496 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:49.642379 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:18:49.642789 19937 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:18:49.681097 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:49.702728 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 44522.3us
I20240501 08:18:49.718768 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:49.770479 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:18:49.773305 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:49.779752 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:49.861266 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:18:49.878842 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:49.899060 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:49.915737 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:49.922628 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:49.926528 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:49.930300 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:49.938879 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:50.009624 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:50.048460 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:50.095155 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:18:50.224535 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:50.228266 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:50.234069 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:50.254456 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:50.255692 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:50.256949 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:50.259763 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:50.282060 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:50.326318 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:50.328861 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:50.353960 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:50.357026 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:50.366778 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:50.372754 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:50.407969 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:50.418552 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:50.456777 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:50.514524 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:50.701718 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:50.708590 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:50.716392 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:50.732079 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:50.736318 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:50.744357 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:50.763408 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:50.763525 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:50.987614 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.020967 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:51.021430 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:51.027734 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:51.029536 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.059660 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:51.092289 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:51.124958 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:18:51.124941 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:51.197916 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.216877 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:51.228565 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:51.231194 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.259048 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:51.274533 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:51.277946 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:51.452333 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:51.454654 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:51.465126 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.476406 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:51.499821 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:51.503206 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.507476 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:51.508457 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:18:51.545410 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:51.560786 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:51.612874 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:51.612903 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:51.701103 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.711293 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:51.732117 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.745136 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:51.745910 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:51.764298 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:51.774910 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:51.812378 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:51.861738 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:51.874133 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:18:51.966729 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:51.969583 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:51.977301 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:51.979338 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:51.995474 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:52.006731 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:52.054566 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:52.055764 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:52.135404 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:52.226145 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:18:52.315709 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:52.317183 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:52.318168 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:52.338995 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:52.345239 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:52.351395 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:52.364356 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:52.390980 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:18:52.399091 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:52.410573 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:52.415817 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:52.424458 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:52.457828 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:18:52.465003 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:52.505386 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:52.523725 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:52.536950 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:52.538856 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:18:52.559345 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 84293us
W20240501 08:18:52.637070 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:52.702469 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:52.754829 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:52.764114 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:52.769085 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:52.803267 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:52.806756 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:18:52.811476 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:52.817338 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:52.824937 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:52.909181 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:53.002530 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:53.027969 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:53.068354 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:18:53.076858 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:53.092790 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:53.120903 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:53.129896 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:53.168043 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:53.178540 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:53.180281 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:53.209574 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:53.223069 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:53.316320 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:53.333611 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:53.349967 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:53.533578 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:53.562853 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:53.571777 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:53.584640 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:53.597107 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:53.606225 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:53.622272 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:53.646139 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:53.653003 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:18:53.752585 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:53.764900 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:18:53.807927 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:53.816200 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:53.820438 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:53.821334 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:18:53.845232 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:53.863534 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:53.890748 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:53.902161 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:53.980492 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:54.041560 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:18:54.046530 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:54.047324 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:54.053102 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:54.062177 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:54.074756 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:18:54.079612 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:54.093670 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:54.125978 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:54.132184 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:54.144483 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:54.194926 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:54.349296 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:54.353113 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:54.379719 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:54.403774 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:54.407785 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:54.425436 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:54.464368 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:54.518630 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:54.555465 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:54.566612 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:54.567559 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:54.589663 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:54.616746 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:54.638126 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:54.681125 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:54.703305 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:54.704159 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:54.722371 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:54.829337 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:54.839395 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:54.931852 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:54.948287 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:18:54.950101 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:54.951253 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:54.974829 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:54.981959 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:55.023854 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:55.024564 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:55.058077 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:55.210362 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:55.244709 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:55.249641 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:55.249981 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:55.251070 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:55.267876 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:55.285039 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:55.289942 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:55.313663 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:55.358482 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:55.416908 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:55.443974 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:55.447721 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:55.461597 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:55.465258 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:55.483106 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:55.550432 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:55.554183 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:18:55.744647 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:55.751433 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:55.761101 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:55.776932 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:55.814532 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:55.814955 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:55.962962 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:55.971324 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:55.981763 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:56.008878 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:56.013365 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:56.040668 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:56.137019 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:56.139053 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:56.208649 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:56.209991 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:56.210798 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:56.233919 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:56.236151 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:56.236295 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:56.251735 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:56.275978 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:56.294508 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:56.331472 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:56.357708 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:56.448271 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:18:56.476224 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:56.497181 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:56.500134 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:56.510833 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:56.516996 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:56.536963 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:56.540172 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 65322.4us
I20240501 08:18:56.559715 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:56.627094 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:56.657696 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:56.662518 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:56.672574 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:56.710733 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:56.711186 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:56.720566 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:56.755923 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:56.784477 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:56.887027 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:18:56.899186 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:18:56.932562 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:56.932564 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:56.945266 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:56.960500 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:56.968959 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:56.975483 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:57.013732 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:57.028620 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:57.070778 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:18:57.222554 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:57.241323 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:57.243499 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:57.260360 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:18:57.268930 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:57.274138 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:57.289170 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:57.293344 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:57.293511 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:57.300220 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:57.386996 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:57.392686 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:18:57.403472 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:57.420691 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:57.438866 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:18:57.446236 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:57.446942 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:57.453572 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:57.477152 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:18:57.481835 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:57.539189 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:18:57.539382 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:57.655912 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:18:57.767920 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:57.773365 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:57.778523 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:57.812405 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:57.839401 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:57.840337 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:57.905743 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:57.912456 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:57.968302 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:57.983685 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:18:58.023859 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:58.043833 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:58.048530 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:58.053666 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:58.067317 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:58.090575 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:18:58.127365 6573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 47887.3us
W20240501 08:18:58.189913 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:58.271811 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:58.282577 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:58.283059 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:58.286684 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:58.308758 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:58.310468 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:58.572952 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:58.582971 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:58.598100 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:58.609400 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:58.609826 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:58.636022 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:58.672926 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:58.689240 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:58.709776 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:18:58.726065 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:58.732601 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:18:58.754760 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:58.762050 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:58.775178 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:58.781536 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:58.804363 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:18:58.807548 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:58.808285 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:18:58.809921 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:58.833197 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:58.869287 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:18:58.994017 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:58.994571 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:59.010308 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:59.015282 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:59.037086 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:59.044607 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:59.108654 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:59.138767 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:18:59.271683 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:18:59.276180 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:59.303063 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:18:59.303082 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:59.309779 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:59.337415 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:59.355116 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:59.367648 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:59.370656 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:59.482705 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:18:59.507746 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:18:59.508854 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:59.512353 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:59.529670 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:59.530989 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:59.543869 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:18:59.545440 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:18:59.605798 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:18:59.616979 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:18:59.702212 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:18:59.713909 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:18:59.854759 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:59.859879 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:18:59.873626 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:18:59.886765 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:18:59.898378 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:18:59.900233 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:18:59.913612 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:59.945132 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:18:59.953912 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:18:59.955358 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:18:59.974834 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:18:59.996773 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:00.014906 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:00.030586 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:00.045010 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:19:00.059227 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:19:00.144846 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:00.185038 21721 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:19:00.185238 21721 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:19:00.185945 21721 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:19:00.187625 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:00.187791 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:19:00.187887 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:19:00.188136 21721 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:19:00.195477 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:19:00.250384 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:00.303164 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:00.304237 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:00.311982 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:00.326541 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:00.327885 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:00.378088 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:00.510003 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:19:00.521229 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:00.568488 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:00.575861 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:00.583053 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:00.594650 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:00.597976 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:00.603859 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:00.609675 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:00.620944 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:00.791368 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:00.794839 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:00.799608 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:00.826737 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:00.832360 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:00.835844 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:00.919070 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:19:00.968724 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:19:01.014410 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:01.040551 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:01.045715 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:01.059726 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:01.074097 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:01.107239 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:01.130613 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:01.147835 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:01.148743 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:01.150761 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 48382.5us
I20240501 08:19:01.179807 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:01.202924 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:01.224673 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:01.248422 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:01.267117 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:01.278589 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:01.282166 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:19:01.311439 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:19:01.449728 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:19:01.470116 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:01.476979 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:01.477787 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:01.483058 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:01.491552 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:01.493014 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:01.508878 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:01.518648 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:01.533519 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:01.753381 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:01.767263 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:01.772013 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:01.780843 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:01.782017 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:01.808562 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:01.896437 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:19:01.897418 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:01.910087 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:19:01.961467 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:01.963285 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:01.974189 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:01.981597 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:01.999434 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:01.999788 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:02.013120 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:02.016076 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:02.039049 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:02.093544 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:19:02.101408 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:19:02.131799 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:02.263446 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:02.276404 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:02.296969 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:02.297053 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:02.304643 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:02.312837 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:02.318799 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:02.347752 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:02.448225 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:02.449034 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:02.449156 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:02.455629 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:02.464694 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:02.465720 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:02.468390 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:02.489225 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:19:02.494385 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:19:02.568882 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 08:19:02.569835 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:02.717298 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:02.739331 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:02.739439 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:02.748761 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:02.796293 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:02.796494 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:02.799170 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:02.846470 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:02.925247 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:02.927282 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:02.951632 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:02.957711 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:02.960086 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:02.961745 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:02.967392 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:03.001497 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:03.153879 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:03.154901 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:03.170783 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:03.171298 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:03.180260 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:03.182745 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:03.189818 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:03.232422 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:03.385699 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:03.387754 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:03.400169 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:03.426898 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:03.430086 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:03.446460 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:03.505502 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:03.521632 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:19:03.558063 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:19:03.571317 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:03.601166 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:03.608371 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:03.625702 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:03.632972 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:03.637385 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:03.640851 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:03.666615 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:19:03.678020 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:03.759330 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:19:03.760358 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:19:03.765978 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:03.923053 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:03.925773 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:03.951272 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:03.965169 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:03.981645 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:04.005595 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:04.013079 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:04.019554 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:04.044225 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:04.057024 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:04.059988 6747 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:04.086248 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:04.099195 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:19:04.108656 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:04.118647 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:04.233408 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:19:04.291566 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:04.395645 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:04.401160 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:04.411765 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:04.417482 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:04.441546 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:04.505167 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:04.511161 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:04.542871 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:04.575939 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:19:04.604876 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:19:04.624640 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:04.723944 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:04.730715 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:04.738505 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:04.784102 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:04.789403 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:04.791580 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:04.807387 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:04.822619 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:19:04.834796 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:04.930184 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:04.958927 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:04.970645 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:04.977806 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:04.982721 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:05.007572 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:05.076671 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:19:05.124264 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:19:05.159288 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:19:05.168602 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:05.178083 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:05.264101 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:05.272310 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:05.286595 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:05.287374 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:05.311244 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:05.318444 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:05.433452 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:05.457895 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:05.467919 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:05.474539 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:05.485697 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:05.502765 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:05.510538 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:05.522297 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:05.709328 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:05.713490 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:05.726023 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:05.729313 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:05.739799 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:05.740550 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:05.762194 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:05.884917 20451 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:19:05.885143 20451 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:19:05.886054 20451 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:19:05.887652 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:19:05.893810 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:19:05.903637 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:19:05.905324 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:05.905515 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:19:05.905807 20451 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:19:05.987357 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:05.999264 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:06.000965 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:06.009361 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:06.014726 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:06.016530 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:06.035351 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:06.047341 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:06.071943 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:06.078014 6702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.3:43505, user_credentials={real_user=slave}} blocked reactor thread for 57122.9us
I20240501 08:19:06.107455 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:06.120838 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:19:06.126992 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:19:06.139652 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:06.141394 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:06.177917 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:06.186375 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:06.246553 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:19:06.255432 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:19:06.329362 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:06.537353 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:19:06.582967 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:06.610641 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:19:06.638147 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:06.647930 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:06.693148 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:06.720876 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:06.731532 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:06.736865 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:06.764832 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:06.787371 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:06.812546 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:06.831617 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:06.832754 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:06.839515 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:06.852524 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:06.864914 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:06.873356 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:06.895179 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:06.897253 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:19:06.898310 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:06.937853 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:07.058652 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:07.060106 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:07.074505 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:07.084559 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:07.091027 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:07.112818 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:07.130338 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:07.199131 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:19:07.314097 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:07.382798 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:07.412267 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:07.415215 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:07.422740 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:07.444558 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:07.445411 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:07.462925 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:07.484124 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:07.492282 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:07.539016 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:07.549110 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:07.567910 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:07.568825 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:07.589366 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:19:07.592650 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:07.603649 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:07.615125 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:07.657665 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:19:07.747514 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:07.773489 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:07.826689 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:07.827857 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:07.829047 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:07.896108 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:07.918998 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:07.920297 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:07.929912 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:08.101718 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:08.122960 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:08.136049 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:08.162326 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:08.197615 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:08.200125 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:08.202561 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:08.206451 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:08.268335 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:08.409188 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:08.414810 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:08.418526 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:08.419121 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:08.430881 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:08.441007 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:08.476540 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:08.488130 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:08.700525 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:08.743747 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:08.757576 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:08.769760 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:08.777019 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:08.781684 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:08.809578 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:08.810583 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:08.836030 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:08.842013 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:08.856042 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:19:08.944373 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:08.950601 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:08.964540 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:08.971527 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:08.973577 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:08.977816 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:09.001773 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:09.005674 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:09.018061 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:09.050040 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:09.192155 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:09.197144 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:09.205508 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:09.220615 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:09.220875 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:09.242576 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:09.243005 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:09.326124 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:09.409816 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:09.444506 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:09.449349 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:19:09.468370 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:09.486018 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:09.488739 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:09.499862 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:09.500839 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:09.506381 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:09.523429 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:09.563545 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:09.622356 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:09.656333 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:09.733853 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:09.742508 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:09.748339 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:09.749317 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:09.751114 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:09.791198 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:09.796695 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:09.875418 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:09.989532 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:09.994338 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:10.017728 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:10.017728 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:10.037181 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:10.047854 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:10.047850 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:10.150041 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:10.181849 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:10.234220 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:10.253485 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:10.256649 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:10.267340 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:10.275332 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:10.297638 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:10.311147 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:10.398806 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:10.435115 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:10.446799 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:10.449349 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:10.453289 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:10.462013 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:10.462564 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:10.485296 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:10.517000 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:10.520156 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:10.526088 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:10.787025 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:10.796886 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:10.802429 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:10.816028 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:10.824649 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:10.838043 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:10.857975 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:10.858422 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:10.885450 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:10.929781 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:10.934688 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:10.938925 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:10.941136 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:11.098981 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:11.139818 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:11.173256 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:11.189821 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:11.203038 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:11.214499 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:11.226794 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:11.238947 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:11.352651 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:11.369271 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:11.414054 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:11.427348 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:11.434579 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:11.460914 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:11.463199 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:19:11.463426 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:11.483691 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:11.501006 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:11.508826 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:11.532785 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:11.587456 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:11.614513 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:11.623850 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:11.623850 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:11.631942 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:11.641000 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:11.651356 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:11.709719 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:11.751353 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:11.933426 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:11.940584 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:11.941632 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:11.967391 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:11.979355 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:12.001693 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:12.002418 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:19:12.003386 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:12.009332 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:12.010977 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:12.031374 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:12.068306 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:12.160634 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:19:12.196264 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:12.214094 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:12.218552 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:12.228601 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:12.240739 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:12.253232 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:12.255558 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:12.264214 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:12.301630 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:12.446115 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:12.464856 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:12.474032 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:12.484544 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:12.486400 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:12.506853 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:12.513428 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:12.612175 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:12.616366 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:19:12.619190 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:12.649251 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:12.715663 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:19:12.772032 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:12.776543 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:12.797552 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:12.799857 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:12.827230 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:12.828871 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:12.857686 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:12.864837 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:12.864982 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:12.876773 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:12.900441 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:12.920296 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:12.938715 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:12.968143 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:12.993398 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:13.034792 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:19:13.196049 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:13.203009 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:13.218636 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:13.230603 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:13.238128 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:13.264983 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:13.306951 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:13.441874 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:13.468134 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:13.470075 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:13.496075 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:13.496917 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:13.527290 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:13.527956 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:13.674865 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:13.684291 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:13.689136 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:13.710848 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:13.728646 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:13.732127 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:13.770889 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:13.793990 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:13.795117 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:13.835214 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:13.859578 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:19:13.946296 23529 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:19:13.946502 23529 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:19:13.947165 23529 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:19:13.949632 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:19:13.949939 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:13.950044 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:19:13.950624 23529 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:19:14.006196 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:14.027063 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:14.035261 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:14.039535 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:14.054376 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:14.073107 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:14.081773 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:14.083403 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:14.129427 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:14.132897 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:14.150178 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:14.157037 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:14.162180 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:14.176374 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:14.178436 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:14.216761 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:14.227223 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:14.235328 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:14.381286 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:14.485865 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:14.486503 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:14.487540 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:14.491858 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:14.492398 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:14.494788 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:14.522325 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:14.522836 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:14.574084 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:14.604719 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:14.632987 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:14.655246 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:19:14.677516 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:14.706717 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:14.760792 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:14.780581 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:14.798590 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:14.799708 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:14.806524 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:14.824383 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:14.826681 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:14.907315 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:14.932472 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:14.958329 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:14.959805 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:14.967713 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:14.981590 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:14.990404 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:15.012682 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:15.054010 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:15.147739 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:15.280521 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:15.309094 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:15.310655 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:15.327783 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:15.331007 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:15.345117 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:15.388466 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:15.418875 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:15.469873 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:15.554536 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:15.568243 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:15.569406 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:15.576505 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:15.582525 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:15.626632 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:15.699754 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:15.747689 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:15.842680 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:15.901149 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:15.925570 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:15.936319 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:16.002535 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:16.003330 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:16.020988 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:16.039827 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:16.044592 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:16.074973 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:16.078516 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:16.108543 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:16.136397 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:16.169000 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:16.206848 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:16.223289 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:16.358736 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:16.361251 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:16.376531 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:16.380517 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:16.385802 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:16.403605 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:16.450191 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:16.473232 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:16.474042 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:16.521957 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:16.616075 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:16.637362 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:16.640539 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:16.683856 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:16.712713 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:16.712985 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:19:16.740487 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:16.741384 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:16.752777 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:16.832221 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:16.849520 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:16.869326 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:16.877297 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:16.877749 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:16.901036 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:16.929201 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:16.946360 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 70077.4us
W20240501 08:19:16.978197 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:19:17.004856 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:19:17.007781 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:17.011991 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:17.070083 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:17.151619 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:17.152405 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:17.163764 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:17.176244 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:17.184137 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:17.208185 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:17.220005 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:17.234525 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:17.251444 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:17.394344 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:17.425825 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:17.435920 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:17.436853 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:17.495744 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:17.522948 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:17.539212 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:17.582840 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:17.583550 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:17.629364 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:17.654760 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:19:17.686836 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:17.704438 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:17.719082 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:17.723381 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:19:17.734027 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:17.746649 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:17.746798 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:17.782748 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:17.811151 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:17.855405 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:17.880899 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:17.908349 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:17.978978 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:17.988888 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:18.008466 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:18.008865 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:18.026859 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:18.042272 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:18.062978 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:18.146112 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:18.180352 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:18.191453 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:18.211704 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:18.218528 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:18.228636 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:18.236462 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:18.253252 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:18.293397 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:19:18.499094 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:18.508314 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:18.529909 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:18.533531 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:18.539273 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:18.542213 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:18.752569 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:18.776232 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:18.801301 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:18.814882 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:18.821264 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:18.836863 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:18.841866 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:18.890630 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:18.911033 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:18.911350 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:18.912606 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:18.923966 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:19:18.959074 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:18.960013 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:18.992229 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:19.007695 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:19.024247 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:19.026355 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:19.065768 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:19:19.065992 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:19.173414 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:19.197517 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:19:19.237501 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:19.253849 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:19.259799 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:19.279650 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:19.282714 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:19:19.284158 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:19.302524 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:19.322163 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:19.380544 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:19.559160 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:19.562027 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:19.585672 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:19:19.586772 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:19.611375 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:19.642855 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:19.647595 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:19.657510 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:19.668653 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:19.691948 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:19.716046 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:19:19.768141 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:19.780205 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:19.787704 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:19.791630 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:19.799306 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:19.799949 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:19.843819 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:19.920637 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:19.953709 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:19.966193 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:20.081423 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:20.100128 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:20.114928 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:20.136286 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:20.168818 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:20.173977 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:20.236294 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:20.252703 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:19:20.274582 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:20.291083 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:20.305835 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:20.315024 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:20.331602 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:20.345387 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:20.346642 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:20.382598 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:20.536231 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:20.545686 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:20.561978 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:20.571645 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:20.579617 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:20.589072 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:20.605077 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:20.633874 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:20.695101 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:20.730537 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:20.816327 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:20.817891 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:20.836014 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:20.866117 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:20.879076 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:20.909392 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:20.913581 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:21.021231 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:21.028051 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:21.059907 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:21.076857 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:21.079145 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:21.082631 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:21.273202 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:21.280418 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:21.305495 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:19:21.369231 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:21.370472 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:21.389786 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:21.390168 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:21.403209 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:21.408217 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:19:21.421917 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:21.473042 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:21.595253 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:21.595789 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:21.614385 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:21.619589 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:21.621402 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:21.640674 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:21.664202 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:21.723584 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:21.753674 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:21.768551 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:21.891932 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:21.892233 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:21.909173 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:21.917461 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:21.929087 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:21.948776 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:21.975607 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:21.978360 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:21.983130 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:22.007098 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:19:22.008587 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:19:22.014385 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:22.016607 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:22.045794 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:22.056804 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:22.056859 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:22.094512 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:22.194065 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:22.194329 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:22.219280 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:19:22.241071 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:22.295199 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:22.297116 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:22.366565 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:22.379972 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:22.381249 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:22.389755 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:22.395156 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:22.403983 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:22.422401 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:22.541595 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:22.570067 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:22.624156 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:22.667661 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:19:22.675658 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:22.695811 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:22.728683 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:22.736312 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:22.753624 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:22.766534 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:22.819449 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:19:22.837699 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:22.848912 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:22.873858 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:22.878822 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:22.886399 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:22.909324 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:19:22.909381 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:22.923991 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:22.994033 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:23.016228 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:23.175529 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:23.178220 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:23.190039 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:23.191182 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:23.199793 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:23.213757 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:23.217820 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:23.317837 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:23.388321 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:23.408752 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:23.439725 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:23.443698 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:23.444900 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:23.477734 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:23.485688 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:23.543633 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:23.545569 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:23.569614 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:23.572320 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:23.572791 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:23.604489 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:23.613668 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:23.624487 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:23.629801 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:23.830423 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:23.840092 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:23.852542 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:23.869694 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:23.884872 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:23.898182 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:23.906971 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:23.908519 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:23.920599 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:23.941023 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:23.954432 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 56 similar messages]
I20240501 08:19:24.101824 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:24.105373 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:24.120091 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:24.121443 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:24.126884 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:24.170671 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:24.201583 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:24.314380 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:24.322525 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:24.341274 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:24.344985 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:24.354705 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:24.359505 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:24.365209 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:24.384146 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:24.462977 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:24.466662 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:24.504504 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:19:24.535971 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:24.656813 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:24.660543 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:24.662204 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:24.664615 25318 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:19:24.664767 25318 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:19:24.665580 25318 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:19:24.674434 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:24.696171 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:24.703017 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 08:19:24.706012 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:24.718096 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:19:24.718281 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:19:24.718528 25318 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:19:24.725262 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:24.766042 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:24.769428 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:24.788540 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:24.803668 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:24.810843 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:19:24.812589 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:24.814971 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:24.826145 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:24.841022 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:24.851444 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:24.898097 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:24.913717 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:19:24.915416 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:25.088035 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:25.097080 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:25.099943 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:25.137915 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:25.140818 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:25.142410 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:25.175228 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:25.204779 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:25.372426 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:25.384367 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:25.395548 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:25.404350 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:25.404994 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:25.412782 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:25.428010 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:25.447062 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:25.520146 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:25.553974 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:25.565420 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:25.577111 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:25.585580 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:25.586176 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:25.588750 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:25.593629 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:25.706892 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:19:25.846812 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:25.883934 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:25.898308 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:25.901556 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:25.915261 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:25.921700 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:25.942823 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:25.942803 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:25.968760 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:26.000366 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:26.040827 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:26.046679 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:26.115561 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:26.126096 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:26.146059 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:26.170105 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:26.305104 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:26.368686 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:26.371099 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:26.372882 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:26.380206 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:26.392843 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:26.402860 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:26.414011 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:26.433300 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:19:26.476703 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:26.580474 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:19:26.650871 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:26.660076 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:26.684262 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:26.701117 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:26.705902 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:26.713698 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:26.727967 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:26.744786 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:26.753765 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:26.774385 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:26.811090 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:26.816110 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:26.889309 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:26.903693 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:26.906286 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:26.918848 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:27.126569 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:19:27.136260 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:27.139075 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:27.145967 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:19:27.162902 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:27.170277 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:19:27.320808 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:27.373975 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:27.379413 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:27.380908 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:27.399467 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:27.400496 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:27.400544 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:27.414597 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:27.420934 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:27.431757 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:27.494860 25318 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:19:27.495070 25318 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:19:27.495234 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:27.495786 25318 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:19:27.500743 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:27.538576 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:27.547780 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:27.571696 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:27.594480 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:27.595350 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:19:27.595537 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:19:27.595844 25318 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:19:27.610834 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:27.671415 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:19:27.716076 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:27.755438 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:19:27.780414 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:27.795872 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:27.796280 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:27.824800 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:27.838212 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:27.843596 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:27.845052 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:19:27.862478 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:27.980105 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:28.000532 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:19:28.024870 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:19:28.034401 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:28.133837 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:28.157325 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:28.159595 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:28.177824 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:28.213268 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:28.222527 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:28.230697 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:28.246361 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:28.259547 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:28.298920 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:28.310374 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:28.328799 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:28.333866 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:28.373986 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:28.453687 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:28.528544 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:28.543541 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:28.546772 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:28.565276 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:28.569329 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:28.571277 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:28.571956 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:28.589767 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:28.597985 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:28.845362 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:28.892154 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:28.906908 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:28.914911 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:28.923985 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:28.942708 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:28.971349 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:28.997895 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:28.998625 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:28.999902 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:29.019587 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:29.027365 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:29.039880 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:19:29.061988 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:29.120666 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:29.157414 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:29.201247 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:29.234123 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:29.295645 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:29.349988 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:29.354293 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:29.362401 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:29.369940 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:29.378083 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:29.395483 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:29.406172 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:19:29.486940 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:29.527359 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:29.545991 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:29.599584 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:29.616355 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:29.619001 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:29.656832 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:29.660066 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:29.666831 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:29.668046 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:29.718667 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:19:29.765913 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:29.770279 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:29.852043 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:29.908485 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:29.928267 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:29.933522 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:29.947306 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:29.949762 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:29.958930 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:29.971206 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:30.109865 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:30.185806 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:30.195807 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:30.199107 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:30.216362 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:30.222281 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:30.224056 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:30.242789 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:30.260360 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:30.335351 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:30.354622 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:30.395597 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:30.446841 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:30.455991 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:30.473434 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:30.484479 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:30.500452 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:30.500887 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:30.566882 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:30.567445 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:30.791898 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:30.807951 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:30.819203 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:30.827411 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:30.875634 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:30.894654 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:30.901402 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:30.921133 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:30.933800 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:30.947221 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:30.976325 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:31.002425 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:31.005919 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:31.028790 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:31.032743 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:31.138569 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:31.249455 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:31.266655 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:31.269510 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:31.283663 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:31.284098 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:31.284289 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:31.309393 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:31.414362 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:31.420931 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:31.560366 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:31.574584 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:31.578512 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:31.583691 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:31.625351 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:31.630511 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:31.651888 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:31.658604 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:31.664651 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:31.677047 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:31.721560 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:31.731513 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:31.743932 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:31.766741 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:31.771268 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:31.776798 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:31.815332 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:31.865615 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:31.898355 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:31.920778 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:31.971426 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:19:31.980674 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:32.004496 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:32.018216 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:32.018604 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:32.023092 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:32.028026 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:32.044518 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:32.091964 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:32.133251 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:19:32.142243 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:32.145376 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:19:32.288961 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:32.292454 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:32.306906 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:32.337774 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:32.355726 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:32.392493 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:32.399871 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:32.430974 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:32.474185 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:32.492544 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:32.504734 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:32.506969 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:32.509788 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:32.517725 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:32.552866 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:32.598408 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:32.626210 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:32.668401 26600 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:19:32.668638 26600 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:19:32.669378 26600 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:19:32.671510 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:32.671792 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:19:32.671933 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:19:32.672231 26600 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:19:32.682371 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:32.693284 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:32.766607 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:19:32.799520 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:32.825826 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:32.832720 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:32.844647 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:32.853901 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:32.891501 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:32.944181 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:19:33.014890 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:33.015480 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:33.037693 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:33.046479 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:33.057914 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:33.093288 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:33.141566 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:33.215855 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:33.224876 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:33.285089 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:33.297978 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:33.306157 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:33.314134 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:33.325394 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:33.328850 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:33.330941 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:33.338764 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:33.350972 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:33.586354 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:33.595075 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:33.609652 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:33.615428 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:33.633577 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:33.637115 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:33.688298 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:33.775405 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:33.819047 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:33.830236 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:33.852756 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:33.874116 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:33.905416 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:33.912817 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:33.916455 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:34.052554 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:19:34.084371 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:34.164686 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:34.177394 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:34.180437 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:34.202165 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:34.211627 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:34.211771 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:34.263845 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:34.264389 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:34.268537 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:34.315585 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:34.316664 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:34.325991 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:34.347479 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:34.373921 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:34.376658 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:34.392742 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:34.396278 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:34.402434 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:34.440981 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:19:34.446031 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:34.452968 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:19:34.469046 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:34.578509 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:34.650888 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:34.663862 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:34.665531 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:34.688169 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:34.689149 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:34.723716 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:34.774412 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:34.941515 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:34.948876 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:34.962498 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:34.975708 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:34.983956 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:34.984366 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:34.991425 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:35.081009 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:35.092605 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:35.092809 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:35.112901 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:35.131902 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:35.135861 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:35.139695 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:35.152524 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:35.152892 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:35.179875 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:35.205152 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:35.231561 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:35.368391 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:35.380575 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:35.384935 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:35.393131 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:35.400516 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:35.416484 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:35.417881 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:35.494774 6572 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 86273us
W20240501 08:19:35.503314 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:35.645673 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:35.677959 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:35.708257 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:35.714196 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:35.729415 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:35.749756 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:35.758539 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:35.774008 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:35.781461 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:35.804610 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:35.913056 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:35.930084 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:35.936941 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:35.978647 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:35.994613 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:35.998157 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:35.998446 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:36.012300 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:36.245857 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:36.247802 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:36.252499 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:36.260722 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:36.277101 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:36.284391 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:36.325707 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:36.493984 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:36.497764 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:36.500329 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:36.510041 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:36.521387 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:36.522202 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:36.562502 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 45478us
W20240501 08:19:36.565989 6574 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 53357.5us
W20240501 08:19:36.571915 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:36.582688 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:36.684959 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:19:36.786092 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:36.803483 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:36.804431 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:36.814168 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:36.817888 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:36.829721 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:36.861698 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:36.868228 7007 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:36.868305 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:36.880240 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:36.887128 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:36.917510 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:36.923895 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:36.935432 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:19:36.936337 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:19:36.957430 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:36.970662 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:36.995234 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:36.996225 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:37.016893 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:37.040377 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:37.148638 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:19:37.239605 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:37.292780 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:37.295465 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:37.298528 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:37.305073 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:37.320423 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:37.324393 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:37.326836 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:19:37.460230 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:37.575608 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:37.585656 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:37.610747 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:37.628242 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:37.648133 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:37.657723 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:37.663225 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:37.683161 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:37.683786 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:37.693213 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:37.747720 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:37.765048 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:37.787658 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:37.793248 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:37.807612 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:37.807775 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:37.810879 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:19:37.814749 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:37.815129 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:37.852622 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:19:37.881091 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:19:37.903839 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:19:37.954952 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:19:37.978010 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:19:38.020495 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:38.023578 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:38.024619 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:38.027161 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:38.039958 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:38.061533 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:38.084728 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:38.202366 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:38.310086 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:38.319631 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:38.330922 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:38.335143 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:38.351850 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:38.368809 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:38.391319 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:38.474352 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:38.526990 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:38.535338 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:38.551137 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:38.552191 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:38.554586 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:38.555467 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:38.800637 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:38.812105 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:38.820844 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:38.825909 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:38.833993 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:38.834208 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:38.866772 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:39.025324 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:39.034452 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:39.039121 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:39.042554 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:39.058794 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:39.059075 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:39.073660 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:39.131088 6703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.6:41219, user_credentials={real_user=slave}} blocked reactor thread for 46974.6us
W20240501 08:19:39.170636 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:19:39.232908 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:19:39.233767 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:39.278983 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:39.279122 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:39.319459 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:39.327486 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:39.334874 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:39.346515 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:39.348533 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:39.360478 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:19:39.373579 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:39.383539 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:39.428171 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:19:39.429060 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:19:39.444177 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:39.563920 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:39.578248 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:39.587977 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:39.609486 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:39.615694 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:39.620218 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:39.634296 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:19:39.678886 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:39.784085 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:39.786753 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:39.792594 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:39.808287 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:39.814604 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:39.819375 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:39.845114 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:39.848606 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:39.877941 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:19:40.053406 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:40.073663 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:40.098894 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:40.124660 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:40.126616 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:40.140560 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:40.152172 6835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 49091.3us
W20240501 08:19:40.158947 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:19:40.162098 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:40.174014 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:40.175874 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:40.208182 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:19:40.284618 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:19:40.293499 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:40.303893 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:40.304589 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:40.304963 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:40.322302 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:40.323276 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:40.355232 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:40.355839 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:40.390990 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:40.401868 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:19:40.548578 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:40.566226 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:40.568611 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:40.574496 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:40.574883 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:40.587466 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:40.791824 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:40.808145 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:40.860224 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:19:40.920300 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:40.925864 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:40.927201 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:40.959967 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:40.962277 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:40.967485 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:19:40.979126 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:40.982184 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:40.999076 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:41.011803 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:41.091259 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:41.126952 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:41.129470 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:41.357743 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:41.371527 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:41.382618 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:41.400907 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:41.411216 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:41.413713 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:41.421093 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:41.570676 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:41.578361 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:19:41.645320 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:41.660362 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:19:41.673238 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:41.673575 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:41.697742 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:19:41.698735 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:41.728250 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:41.745841 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:41.747154 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:41.770562 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:41.791426 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:41.795775 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:41.828330 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:41.834239 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:41.842240 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:41.856540 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:19:41.872609 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:41.904186 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:41.954193 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:41.985322 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:41.992487 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:42.200275 6574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:19:42.202466 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:42.213531 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:19:42.284618 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:42.316205 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:42.337975 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:42.344929 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:42.358425 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:42.378055 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:42.409797 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:19:42.428442 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:42.458056 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:42.466830 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:42.479655 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:42.502177 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:42.509291 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:42.553329 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:42.553894 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:42.626564 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:42.701807 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:19:42.766013 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:42.783007 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:42.785938 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:42.791514 25318 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:19:42.791682 25318 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:19:42.792487 25318 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:19:42.798379 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:42.809376 8895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20240501 08:19:42.814189 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:42.814589 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:19:42.818959 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:42.822469 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:42.822615 8892 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:19:42.823073 25318 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:19:42.899173 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:19:42.900544 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:42.940164 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:42.947901 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:42.968118 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:42.969295 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:42.973351 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:42.974191 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:43.037767 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:43.055123 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:43.055536 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:43.072474 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:43.099915 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:43.121536 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:43.132627 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:19:43.207877 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:19:43.263317 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:43.275221 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:43.279250 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:43.288897 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:43.304936 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:43.311904 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:43.326406 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:43.437570 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:43.501014 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:43.546919 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:43.573290 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:43.588493 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:43.588789 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:43.597307 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:43.628580 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:43.782354 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:43.783501 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:43.803596 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:43.822734 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:43.826393 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:43.863642 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:43.977257 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:44.037130 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:19:44.045185 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:44.047614 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:44.051467 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:44.063473 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:44.080629 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:44.090647 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:44.095268 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:44.108263 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:44.268954 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:19:44.320202 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:44.330363 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:44.354548 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:44.367024 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:44.375447 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:44.399811 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:19:44.400453 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:44.407951 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:44.462659 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:44.468029 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:44.483433 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:19:44.495250 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:44.496191 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:44.512863 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:44.513644 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:44.527408 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:44.533615 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:44.546958 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:44.566351 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:44.579429 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:44.624727 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:44.725262 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:44.773473 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:44.785274 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:44.798607 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:44.834156 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:44.839489 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:44.845276 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:44.859158 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:45.024873 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:45.030606 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:19:45.034652 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:45.040784 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:45.044270 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:45.044792 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:45.065924 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:45.097888 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:45.118513 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:45.131202 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:45.214919 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:45.219996 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:45.275291 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:45.295925 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:45.301824 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:45.316195 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:45.318060 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:45.338686 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:45.350482 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:45.444617 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:45.474300 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:45.629628 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:45.634012 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:19:45.643579 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:45.644557 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:45.646608 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:45.658609 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:45.681382 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:45.686493 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:19:45.698448 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:45.703235 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:45.727720 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:45.745229 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:45.772087 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:45.797034 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:45.821509 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:45.827422 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:45.866642 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:46.041671 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:46.104393 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:46.110733 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:46.122984 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:46.135905 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:46.137611 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:46.148020 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:46.236276 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:46.344964 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:46.400421 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:46.420192 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:46.421502 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:46.444094 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:46.468788 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:46.480053 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:46.501077 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:46.528658 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:46.540957 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:46.576084 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:46.600283 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:46.604478 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:46.666811 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:46.681685 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:46.720289 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:46.809420 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:46.811641 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:46.814651 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:46.818045 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:46.824127 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:46.839267 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:46.855926 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:46.874390 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:19:46.917058 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:47.043932 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:19:47.057195 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:47.086727 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:47.086979 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:47.093444 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:47.107681 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:47.127059 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:47.132395 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:19:47.150918 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:47.158737 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:47.183658 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:47.205348 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:47.220419 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:47.248957 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:47.258194 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:47.265055 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 08:19:47.275342 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:47.309177 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:47.324661 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:47.332114 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:47.381129 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:47.431028 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:47.499104 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:47.507534 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:47.537425 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:47.549058 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:47.574309 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:47.657817 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:19:47.664654 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:47.679632 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:47.713037 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:47.716611 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:47.776980 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:47.827463 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:47.832798 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:47.836630 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:47.844370 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:47.862277 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:47.898305 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:48.020304 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:19:48.058740 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:19:48.073024 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:48.141239 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:48.168318 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:48.173050 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:19:48.179304 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:48.182065 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:48.188308 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:48.197798 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:48.200875 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:48.201790 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:48.293594 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:48.418956 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:48.432369 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:48.458420 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:48.481968 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:48.505544 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:48.521373 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:48.522514 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:48.532842 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:48.546378 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:48.560978 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:48.562727 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:48.573073 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:48.583489 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:48.613179 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:48.639562 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:48.640259 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:48.808666 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:19:48.884775 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:48.902937 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:48.903103 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:48.927718 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:48.930960 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:48.939123 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:48.944013 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:49.081044 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:49.142388 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:49.212623 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:49.247229 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:49.250056 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:49.277926 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:49.320283 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:49.329517 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:49.331058 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:49.351486 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:19:49.353560 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:49.381466 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:49.383970 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:49.389653 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:49.416718 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:49.423907 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:19:49.439486 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:49.450809 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:49.462721 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:49.479704 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:49.578752 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:49.583361 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:19:49.647653 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:49.650030 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:19:49.696409 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:49.716665 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:49.730929 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:49.760222 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:49.784247 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:49.797188 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:49.824213 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:49.888020 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:50.026549 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:50.031661 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:50.049383 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:50.055894 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:50.062716 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:50.073570 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:50.148828 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:50.166085 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:50.180534 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:50.228369 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:50.257036 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:50.258396 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:50.261485 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:50.261974 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:50.265676 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:50.272073 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:50.299404 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:50.337364 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:50.522784 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:50.523380 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:50.523651 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:50.547797 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:50.562984 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:50.596652 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:50.627142 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:50.630709 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:50.752393 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:50.768301 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:50.772810 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:50.772981 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:50.774065 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:50.777845 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:50.795840 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:50.926717 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:50.947516 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:50.982875 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:19:50.994398 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:51.028872 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:19:51.035916 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:51.056620 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:51.061669 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:51.083925 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:51.087210 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:51.098325 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:51.136798 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:51.263273 29081 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:19:51.263494 29081 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:19:51.264199 29081 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:19:51.270344 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:51.284138 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:19:51.284374 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:19:51.289393 29081 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:19:51.330935 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:51.357656 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:51.367784 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:51.391928 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:51.437846 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:51.445787 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:51.446072 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:51.505174 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:51.553139 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:51.562130 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:51.592694 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:51.602823 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:51.603983 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:51.640987 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:51.652632 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:51.664362 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:19:51.776633 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:51.802942 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:51.855751 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:51.864449 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:51.864457 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:51.894115 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:51.898795 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:51.922533 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:51.927222 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:51.938885 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:51.946029 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:19:52.019232 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:19:52.034559 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:52.075706 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:52.105331 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:52.110482 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:52.138125 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:52.148027 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:52.167557 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:52.187878 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:52.192570 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:52.222831 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:52.343736 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:52.350414 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:52.363626 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:52.375928 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:52.376227 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:52.379885 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:52.406751 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:52.431567 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:19:52.643813 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:52.650727 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:52.660427 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:52.663300 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:52.681316 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:52.685313 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:52.713922 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:52.715145 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:52.745772 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:52.746222 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:52.805320 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:52.833670 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:52.837284 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:52.866791 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:52.872918 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:52.879928 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:52.937618 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:52.937683 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:53.048969 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:53.115235 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:53.127590 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:53.153388 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:53.165913 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:53.175072 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:53.180167 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:53.225508 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:19:53.233072 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 08:19:53.289196 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:53.357332 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:53.394903 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:53.445931 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:53.451085 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:53.473690 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:53.480921 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:53.483884 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:53.486702 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:53.507105 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:53.511524 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:53.517125 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:53.526947 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:53.549700 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:53.557866 8892 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.3:43505, user_credentials={real_user=slave}} blocked reactor thread for 58202.8us
I20240501 08:19:53.582217 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:53.588404 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:53.617316 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:53.620563 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:53.620888 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:53.779806 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:53.848884 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:54.095232 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:54.166679 8939 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:54.167078 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:54.177153 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:54.181288 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:54.187572 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:54.201788 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:54.215164 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:54.239421 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:54.241325 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:54.259562 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:54.265347 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:54.280817 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:54.288395 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:19:54.300546 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:54.305583 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:54.319883 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:54.322037 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:54.378196 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:54.414805 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:54.422580 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:54.424149 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:19:54.433243 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 56 similar messages]
I20240501 08:19:54.560860 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:54.571993 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:54.579051 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:54.591117 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:54.593561 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:54.594326 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:54.624964 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:54.704977 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:54.745472 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:54.846513 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:54.859988 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:54.864885 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:54.866550 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:54.870328 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:54.935555 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:54.983904 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:55.029014 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:55.029706 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:55.030598 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:55.086380 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:55.087913 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:55.102629 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:55.111240 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:55.115340 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:55.244841 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:55.272289 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:55.352345 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:55.364181 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:55.366086 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:55.371937 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:55.387516 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:55.396123 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:55.451237 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:55.478709 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:55.566346 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:55.585580 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:55.589902 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:55.602129 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:55.631392 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:55.636016 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:55.652807 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:55.672199 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:55.711129 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:55.817194 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:55.818364 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:55.831869 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:55.844295 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:19:55.845930 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:55.853018 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:55.860101 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:55.930130 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:55.933602 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:56.011422 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:56.030817 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:56.088176 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:56.113502 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:56.120321 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:56.167315 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:56.179981 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:56.202924 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:56.243542 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:19:56.245982 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:56.298300 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:56.314934 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:56.319540 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:56.340078 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:56.340955 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:56.378356 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:56.394089 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:56.399677 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:56.504325 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:56.549438 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:56.551409 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:56.563393 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:56.566514 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:56.586773 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:56.613292 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:56.620625 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:56.827623 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:56.832331 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:56.840160 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:56.850129 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:56.856350 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:56.859594 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:56.895828 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:56.904624 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:19:56.969417 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:56.975363 6835 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:19:56.979992 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:19:57.011895 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:19:57.015667 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:19:57.086069 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:57.090883 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:57.099005 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:57.105091 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:57.114532 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:57.146616 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:57.176348 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:57.228377 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:19:57.405635 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:57.418555 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:57.432554 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:57.459329 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:57.479199 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:57.523371 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:57.536305 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:57.539170 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:57.563689 6835 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:19:57.581133 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:57.585583 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:19:57.586616 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:57.593232 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:57.595764 6574 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:19:57.614900 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:57.646538 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:57.665315 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:57.745280 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:19:57.822294 6965 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:19:57.835129 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:57.843036 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:57.843366 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:57.855441 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:57.879495 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:57.889070 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:57.997376 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:19:58.074095 6574 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:19:58.117242 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:58.145116 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:58.153882 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:58.187899 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:58.209384 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:58.212111 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:58.219699 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:58.274771 6965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:19:58.355890 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:58.356302 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:19:58.359604 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:19:58.369868 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:58.372749 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:58.399879 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:58.400758 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 08:19:58.473433 6835 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:19:58.474845 6835 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:19:58.478924 6965 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:19:58.528900 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:19:58.534583 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:58.549165 30163 raft_consensus.cc:998] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.052s)
I20240501 08:19:58.550029 30163 consensus_queue.cc:237] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:19:58.552657 8693 raft_consensus.cc:1261] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:19:58.553198 30522 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:19:58.553329 7027 raft_consensus.cc:1261] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:19:58.553869 30522 consensus_queue.cc:1035] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:19:58.559705 30522 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:58.563057 8693 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:58.563390 7027 raft_consensus.cc:2932] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:19:58.566792 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:58.569490 6492 catalog_manager.cc:5484] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:58.589602 30532 raft_consensus.cc:998] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.106s)
I20240501 08:19:58.590415 30532 consensus_queue.cc:237] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:19:58.593398 30533 raft_consensus.cc:998] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.149s)
I20240501 08:19:58.599980 8694 raft_consensus.cc:1261] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:19:58.600661 30408 consensus_queue.cc:1035] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:19:58.600898 30533 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:19:58.609493 8958 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:19:58.610831 30409 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240501 08:19:58.616107 8693 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:58.622416 30408 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:19:58.623384 6484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:19:58.624142 30522 raft_consensus.cc:998] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.154s)
I20240501 08:19:58.624814 30522 consensus_queue.cc:237] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:19:58.626611 30539 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:19:58.628137 8958 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:19:58.629262 7027 raft_consensus.cc:1261] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:19:58.629864 30163 consensus_queue.cc:1035] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:19:58.634395 6490 catalog_manager.cc:5484] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 reported cstate change: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:58.642061 30163 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:19:58.646977 7027 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
W20240501 08:19:58.654232 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:58.664472 6896 consensus_queue.cc:237] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
W20240501 08:19:58.667346 6834 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): Couldn't send request to peer 28b7613bd4834f3ab245c82dd7e80240. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:19:58.667357 8694 raft_consensus.cc:1261] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:19:58.668210 30539 consensus_queue.cc:1035] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:19:58.669016 6495 catalog_manager.cc:5484] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:19:58.671583 30539 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:19:58.675451 8694 raft_consensus.cc:2932] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:19:58.678093 6497 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 8 to 9, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:19:58.682922 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6575b94bd51546f1bd3bb862f012e7ca with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:19:58.683415 6896 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
W20240501 08:19:58.685603 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:19:58.690040 8958 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:19:58.691226 30539 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:19:58.687497 6497 catalog_manager.cc:5484] T 6575b94bd51546f1bd3bb862f012e7ca P aba7c226aaca4e8c90931f547d48aaa3 reported cstate change: config changed from index 8 to 9, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:19:58.695791 30539 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:19:58.699172 8957 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:19:58.712100 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5f090fda814f455a88dccd6680bdd05f with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:19:58.721710 6637 consensus_queue.cc:237] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:19:58.724334 6497 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 9 to 10, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
W20240501 08:19:58.726948 6572 consensus_peers.cc:485] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4db7f887af8f425c8fa661196add4d37. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:19:58.729475 7027 raft_consensus.cc:1261] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:19:58.730700 30522 consensus_queue.cc:1035] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:19:58.731526 6832 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): Couldn't send request to peer 97cdee0bf9d84645a1451f9822c5d699. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:19:58.734864 7027 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:58.739612 30488 raft_consensus.cc:998] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.322s)
I20240501 08:19:58.740520 30488 consensus_queue.cc:237] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:19:58.734023 30163 raft_consensus.cc:2932] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:58.756852 6495 catalog_manager.cc:5484] T 4db7f887af8f425c8fa661196add4d37 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 8 to 9, VOTER aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:58.761881 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4db7f887af8f425c8fa661196add4d37 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240501 08:19:58.766553 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:19:58.767524 8958 raft_consensus.cc:1261] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:19:58.768218 30026 consensus_queue.cc:1035] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:19:58.776371 30026 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:19:58.782454 8958 raft_consensus.cc:2932] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:19:58.791014 6497 catalog_manager.cc:5484] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 8 to 9, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:19:58.831959 30522 raft_consensus.cc:998] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.162s)
I20240501 08:19:58.832682 30522 consensus_queue.cc:237] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
W20240501 08:19:58.837543 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:19:58.839035 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:58.840865 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:58.848811 30292 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:19:58.849028 30292 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:19:58.849777 30292 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:19:58.855895 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:19:58.856397 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:58.856510 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:19:58.857197 30292 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:19:58.913717 30026 raft_consensus.cc:998] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.114s)
I20240501 08:19:58.914665 30026 consensus_queue.cc:237] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } }
I20240501 08:19:58.920192 8958 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:19:58.924417 8694 raft_consensus.cc:1261] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 3 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 08:19:58.925345 30522 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:19:58.925761 30026 consensus_queue.cc:1035] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20240501 08:19:58.926946 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:19:58.933598 30026 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:19:58.941517 8694 raft_consensus.cc:2932] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:19:58.946529 30163 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:19:58.948316 6497 catalog_manager.cc:5484] T a25e1ddb987b4f8c8a4031053f581c7c P aba7c226aaca4e8c90931f547d48aaa3 reported cstate change: config changed from index 10 to 11, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 3 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } }
I20240501 08:19:58.953218 6490 catalog_manager.cc:5484] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:19:58.954643 8958 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:19:58.958818 6637 consensus_queue.cc:237] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:19:58.961967 8958 raft_consensus.cc:1261] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 08:19:58.961966 6574 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): Couldn't send request to peer cd3434d87b3c46c6848a711cdf6b150c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:19:58.962574 30163 consensus_queue.cc:1035] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:19:58.965761 30163 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:19:58.971493 8957 raft_consensus.cc:2932] T 491674dfbf74498e8a787bebaa5e95da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:19:58.974859 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 491674dfbf74498e8a787bebaa5e95da with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:19:58.979815 30408 raft_consensus.cc:998] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.327s)
I20240501 08:19:58.980302 30533 raft_consensus.cc:998] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.491s)
I20240501 08:19:58.980468 30408 consensus_queue.cc:237] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:19:58.980949 30533 consensus_queue.cc:237] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:19:58.982579 6490 catalog_manager.cc:5484] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:19:58.999097 7027 raft_consensus.cc:1261] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:19:58.999298 8694 raft_consensus.cc:1261] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:19:58.999992 30539 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:19:59.000701 30532 consensus_queue.cc:1035] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:19:59.004791 8957 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:19:59.005653 30408 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:19:59.007272 30532 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:59.007879 8694 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:59.009217 30409 raft_consensus.cc:998] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.366s)
I20240501 08:19:59.009830 30409 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:19:59.011847 7027 raft_consensus.cc:2932] T cfc0c7b429754d92b4dd166c073ac8ed P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:59.013693 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:19:59.018123 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:19:59.023532 6769 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 08:19:59.024169 30408 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:19:59.028183 30532 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:19:59.029024 6769 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:19:59.036435 6497 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 11 to 12, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
W20240501 08:19:59.040578 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:59.043977 6896 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:19:59.046434 6768 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
W20240501 08:19:59.047508 6832 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): Couldn't send request to peer 97cdee0bf9d84645a1451f9822c5d699. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:19:59.047565 6490 catalog_manager.cc:5484] T cfc0c7b429754d92b4dd166c073ac8ed P aba7c226aaca4e8c90931f547d48aaa3 reported cstate change: config changed from index 6 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:19:59.052083 30408 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:19:59.058506 30409 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:19:59.059492 6768 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:19:59.064262 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:19:59.065874 6497 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 12 to 13, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:19:59.079042 30408 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:19:59.084731 8957 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:19:59.100594 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:19:59.108325 6490 catalog_manager.cc:5484] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 7 to 8, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:19:59.116933 30590 ts_tablet_manager.cc:902] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:19:59.134246 6896 consensus_queue.cc:237] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
W20240501 08:19:59.137715 6832 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): Couldn't send request to peer 97cdee0bf9d84645a1451f9822c5d699. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:19:59.138399 8957 raft_consensus.cc:1261] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:19:59.138734 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:19:59.139199 30408 consensus_queue.cc:1035] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:19:59.142241 30532 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:19:59.146500 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:19:59.147231 8958 raft_consensus.cc:2932] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:19:59.156630 30590 tablet_copy_client.cc:308] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
W20240501 08:19:59.157367 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:19:59.157765 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 5f090fda814f455a88dccd6680bdd05f from peer 97cdee0bf9d84645a1451f9822c5d699 ({username='slave'} at 127.25.107.1:37175)
I20240501 08:19:59.157943 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 5f090fda814f455a88dccd6680bdd05f from peer 97cdee0bf9d84645a1451f9822c5d699 at {username='slave'} at 127.25.107.1:37175: session id = 97cdee0bf9d84645a1451f9822c5d699-5f090fda814f455a88dccd6680bdd05f
I20240501 08:19:59.162089 6919 tablet_copy_source_session.cc:205] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:19:59.164192 30590 ts_tablet_manager.cc:1875] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:19:59.173477 30590 ts_tablet_manager.cc:1888] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:19:59.177315 30590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f090fda814f455a88dccd6680bdd05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:19:59.178062 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:59.187000 30590 tablet_copy_client.cc:783] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Starting download of 5 data blocks...
I20240501 08:19:59.196683 30590 tablet_copy_client.cc:647] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Starting download of 1 WAL segments...
I20240501 08:19:59.199891 30590 tablet_copy_client.cc:523] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:19:59.205973 30590 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:19:59.215935 30602 ts_tablet_manager.cc:902] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:19:59.218483 6497 catalog_manager.cc:5484] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 8 to 9, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New cstate: current_term: 1 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:19:59.241672 30590 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:19:59.242152 30590 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap complete.
W20240501 08:19:59.242161 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:19:59.242522 30590 ts_tablet_manager.cc:1367] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.037s user 0.035s sys 0.000s
I20240501 08:19:59.243451 30590 raft_consensus.cc:348] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:19:59.243727 30590 raft_consensus.cc:729] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:19:59.244024 30590 consensus_queue.cc:260] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:19:59.244793 30590 ts_tablet_manager.cc:1398] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:19:59.248013 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 97cdee0bf9d84645a1451f9822c5d699-5f090fda814f455a88dccd6680bdd05f received from {username='slave'} at 127.25.107.1:37175
I20240501 08:19:59.248158 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 97cdee0bf9d84645a1451f9822c5d699-5f090fda814f455a88dccd6680bdd05f on tablet 5f090fda814f455a88dccd6680bdd05f with peer 97cdee0bf9d84645a1451f9822c5d699
I20240501 08:19:59.282675 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:59.309283 30602 tablet_copy_client.cc:308] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:19:59.316900 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 6575b94bd51546f1bd3bb862f012e7ca from peer 28b7613bd4834f3ab245c82dd7e80240 ({username='slave'} at 127.25.107.4:38861)
I20240501 08:19:59.317139 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 6575b94bd51546f1bd3bb862f012e7ca from peer 28b7613bd4834f3ab245c82dd7e80240 at {username='slave'} at 127.25.107.4:38861: session id = 28b7613bd4834f3ab245c82dd7e80240-6575b94bd51546f1bd3bb862f012e7ca
I20240501 08:19:59.339337 30611 ts_tablet_manager.cc:902] T 4db7f887af8f425c8fa661196add4d37 P aba7c226aaca4e8c90931f547d48aaa3: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
F20240501 08:19:59.339776 30611 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.queue_time_us, tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.queue_time_us) (tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.run_cpu_time_us, tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.run_cpu_time_us) (tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.run_wall_time_us, tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.run_wall_time_us) (tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.queue_time_us, tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.queue_time_us) (tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_cpu_time_us, tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_cpu_time_us) (tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_wall_time_us, tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_wall_time_us) (tablet-download-pool-29506d751571401f86bced7e6f5415d7.queue_time_us, tablet-download-pool-29506d751571401f86bced7e6f5415d7.queue_time_us) (tablet-download-pool-29506d751571401f86bced7e6f5415d7.run_cpu_time_us, tablet-download-pool-29506d751571401f86bced7e6f5415d7.run_cpu_time_us) (tablet-download-pool-29506d751571401f86bced7e6f5415d7.run_wall_time_us, tablet-download-pool-29506d751571401f86bced7e6f5415d7.run_wall_time_us) (tablet-download-pool-358b6fe9c68349cb8cb963bb3a9ed2e9.queue_time_us, tablet-download-pool-358b6fe9c68349cb8cb963bb3a9ed2e9.queue_time_us) (tablet-download-pool-358b6fe9c68349cb8cb963bb3a9ed2e9.run_cpu_time_us, tablet-download-pool-358b6fe9c68349cb8cb963bb3a9ed2e9.run_cpu_time_us) (tablet-download-pool-358b6fe9c68349cb8cb963bb3a9ed2e9.run_wall_time_us, tablet-download-pool-358b6fe9c68349cb8cb963bb3a9ed2e9.run_wall_time_us) (tablet-download-pool-37a6157d7d734a80900139944481928c.queue_time_us, tablet-download-pool-37a6157d7d734a80900139944481928c.queue_time_us) (tablet-download-pool-37a6157d7d734a80900139944481928c.run_cpu_time_us, tablet-download-pool-37a6157d7d734a80900139944481928c.run_cpu_time_us) (tablet-download-pool-37a6157d7d734a80900139944481928c.run_wall_time_us, tablet-download-pool-37a6157d7d734a80900139944481928c.run_wall_time_us) (tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.queue_time_us, tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.queue_time_us) (tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.run_cpu_time_us, tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.run_cpu_time_us) (tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.run_wall_time_us, tablet-download-pool-45d076303ea24d7fa0fed5a9d132019c.run_wall_time_us) (tablet-download-pool-4db7f887af8f425c8fa661196add4d37.queue_time_us, tablet-download-pool-4db7f887af8f425c8fa661196add4d37.queue_time_us) (tablet-download-pool-60238162349647c681685349c6f8ede4.queue_time_us, tablet-download-pool-60238162349647c681685349c6f8ede4.queue_time_us) (tablet-download-pool-60238162349647c681685349c6f8ede4.run_cpu_time_us, tablet-download-pool-60238162349647c681685349c6f8ede4.run_cpu_time_us) (tablet-download-pool-60238162349647c681685349c6f8ede4.run_wall_time_us, tablet-download-pool-60238162349647c681685349c6f8ede4.run_wall_time_us) (tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.queue_time_us, tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.queue_time_us) (tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.run_cpu_time_us, tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.run_cpu_time_us) (tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.run_wall_time_us, tablet-download-pool-6575b94bd51546f1bd3bb862f012e7ca.run_wall_time_us) (tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.queue_time_us, tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.queue_time_us) (tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.run_cpu_time_us, tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.run_cpu_time_us) (tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.run_wall_time_us, tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.run_wall_time_us) (tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.queue_time_us, tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.queue_time_us) (tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.run_cpu_time_us, tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.run_cpu_time_us) (tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.run_wall_time_us, tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.run_wall_time_us) (tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.queue_time_us, tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.queue_time_us) (tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.run_cpu_time_us, tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.run_cpu_time_us) (tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.run_wall_time_us, tablet-download-pool-9b9bd7500e8747c6b8ee94a1faa1c5f7.run_wall_time_us) (tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.queue_time_us, tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.queue_time_us) (tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_cpu_time_us, tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_cpu_time_us) (tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_wall_time_us, tablet-download-pool-a25e1ddb987b4f8c8a4031053f581c7c.run_wall_time_us) (tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.queue_time_us, tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.queue_time_us) (tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.run_cpu_time_us, tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.run_cpu_time_us) (tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.run_wall_time_us, tablet-download-pool-cfc0c7b429754d92b4dd166c073ac8ed.run_wall_time_us) (tablet-download-pool-dc19d80bad46460087c143778a4b0638.queue_time_us, tablet-download-pool-dc19d80bad46460087c143778a4b0638.queue_time_us) (tablet-download-pool-dc19d80bad46460087c143778a4b0638.run_cpu_time_us, tablet-download-pool-dc19d80bad46460087c143778a4b0638.run_cpu_time_us) (tablet-download-pool-dc19d80bad46460087c143778a4b0638.run_wall_time_us, tablet-download-pool-dc19d80bad46460087c143778a4b0638.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f76d2ba2dcd google::LogMessage::Fail() at ??:0
@ 0x7f76d2ba6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f76d2ba27cc google::LogMessage::Flush() at ??:0
I20240501 08:19:59.363230 6919 tablet_copy_source_session.cc:205] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 5 blocks and 1 log segments
@ 0x7f76d2ba3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 08:19:59.364830 30602 ts_tablet_manager.cc:1875] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:19:59.374344 30602 ts_tablet_manager.cc:1888] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 08:19:59.381330 30602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6575b94bd51546f1bd3bb862f012e7ca. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f76d58e3742 kudu::TraceMetrics::InternName() at ??:0
I20240501 08:19:59.401369 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:59.408638 30602 tablet_copy_client.cc:783] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Starting download of 5 data blocks...
@ 0x7f76d58aff8b kudu::ThreadPool::ThreadPool() at ??:0
I20240501 08:19:59.420378 30602 tablet_copy_client.cc:647] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Starting download of 1 WAL segments...
@ 0x7f76d58a9cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 08:19:59.428265 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:59.430936 8675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f76e4cfc8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f76e4d14daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 08:19:59.442785 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(5f090fda814f455a88dccd6680bdd05f): 4834 bytes on disk
I20240501 08:19:59.443457 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(5f090fda814f455a88dccd6680bdd05f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
@ 0x7f76e4e3f480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 08:19:59.452450 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7f76e4e77a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f76e4e5a981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f76e4e5a70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f76d58b3b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f76d58b719c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f76d58b7019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f76d589201e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f76d8c176db start_thread at ??:0
I20240501 08:19:59.481748 30602 tablet_copy_client.cc:523] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 08:19:59.481748 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
@ 0x7f76d1a5571f clone at ??:0
I20240501 08:19:59.483839 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:19:59.496510 30602 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:19:59.503547 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:19:59.537992 30602 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:19:59.538519 30602 tablet_bootstrap.cc:492] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap complete.
I20240501 08:19:59.538863 30602 ts_tablet_manager.cc:1367] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.014s
I20240501 08:19:59.539714 30602 raft_consensus.cc:348] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:19:59.539974 30602 raft_consensus.cc:729] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:19:59.540305 30602 consensus_queue.cc:260] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:19:59.541731 30602 ts_tablet_manager.cc:1398] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:19:59.542475 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 28b7613bd4834f3ab245c82dd7e80240-6575b94bd51546f1bd3bb862f012e7ca received from {username='slave'} at 127.25.107.4:38861
I20240501 08:19:59.542618 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 28b7613bd4834f3ab245c82dd7e80240-6575b94bd51546f1bd3bb862f012e7ca on tablet 6575b94bd51546f1bd3bb862f012e7ca with peer 28b7613bd4834f3ab245c82dd7e80240
W20240501 08:19:59.547627 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:59.551705 30633 ts_tablet_manager.cc:902] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:19:59.559718 30590 ts_tablet_manager.cc:902] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
W20240501 08:19:59.575954 6571 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:19:59.588202 6835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240501 08:19:59.624091 6962 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:19:59.624272 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:19:59.633827 30633 tablet_copy_client.cc:308] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:19:59.643190 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 491674dfbf74498e8a787bebaa5e95da from peer cd3434d87b3c46c6848a711cdf6b150c ({username='slave'} at 127.25.107.2:46687)
I20240501 08:19:59.643378 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 491674dfbf74498e8a787bebaa5e95da from peer cd3434d87b3c46c6848a711cdf6b150c at {username='slave'} at 127.25.107.2:46687: session id = cd3434d87b3c46c6848a711cdf6b150c-491674dfbf74498e8a787bebaa5e95da
I20240501 08:19:59.647260 6658 tablet_copy_source_session.cc:205] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:19:59.649956 6637 raft_consensus.cc:1203] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 08:19:59.650568 7027 raft_consensus.cc:1203] T 6575b94bd51546f1bd3bb862f012e7ca P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 08:19:59.650990 30633 ts_tablet_manager.cc:1875] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:19:59.676111 30590 tablet_copy_client.cc:308] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:19:59.677003 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer 97cdee0bf9d84645a1451f9822c5d699 ({username='slave'} at 127.25.107.1:37175)
I20240501 08:19:59.677201 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer 97cdee0bf9d84645a1451f9822c5d699 at {username='slave'} at 127.25.107.1:37175: session id = 97cdee0bf9d84645a1451f9822c5d699-30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:19:59.680794 6919 tablet_copy_source_session.cc:205] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:19:59.682519 30590 ts_tablet_manager.cc:1875] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:19:59.691573 30590 ts_tablet_manager.cc:1888] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 08:19:59.691879 30590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4f075057e42d0bb3f8cdabdfbc643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:19:59.695736 30590 tablet_copy_client.cc:783] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Starting download of 5 data blocks...
I20240501 08:19:59.702440 30590 tablet_copy_client.cc:647] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Starting download of 1 WAL segments...
I20240501 08:19:59.702775 30656 ts_tablet_manager.cc:902] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:19:59.703047 30633 ts_tablet_manager.cc:1888] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 08:19:59.703318 30633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491674dfbf74498e8a787bebaa5e95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:19:59.704447 30590 tablet_copy_client.cc:523] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:19:59.709233 30590 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:19:59.730957 30633 tablet_copy_client.cc:783] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 5 data blocks...
I20240501 08:19:59.738536 30633 tablet_copy_client.cc:647] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:19:59.740694 30633 tablet_copy_client.cc:523] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:19:59.747867 30633 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:19:59.752684 30590 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:19:59.753157 30590 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap complete.
I20240501 08:19:59.753451 30590 ts_tablet_manager.cc:1367] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.044s user 0.019s sys 0.015s
I20240501 08:19:59.754246 30590 raft_consensus.cc:348] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:19:59.754494 30590 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:19:59.754736 30590 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:19:59.755260 30590 ts_tablet_manager.cc:1398] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:19:59.756781 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 97cdee0bf9d84645a1451f9822c5d699-30b4f075057e42d0bb3f8cdabdfbc643 received from {username='slave'} at 127.25.107.1:37175
I20240501 08:19:59.756932 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 97cdee0bf9d84645a1451f9822c5d699-30b4f075057e42d0bb3f8cdabdfbc643 on tablet 30b4f075057e42d0bb3f8cdabdfbc643 with peer 97cdee0bf9d84645a1451f9822c5d699
I20240501 08:19:59.771960 30656 tablet_copy_client.cc:308] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:19:59.772696 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 from peer 97cdee0bf9d84645a1451f9822c5d699 ({username='slave'} at 127.25.107.1:37175)
I20240501 08:19:59.772862 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 from peer 97cdee0bf9d84645a1451f9822c5d699 at {username='slave'} at 127.25.107.1:37175: session id = 97cdee0bf9d84645a1451f9822c5d699-399d1a392b3f4d9e8531b6e0b5ad2a61
I20240501 08:19:59.776429 6919 tablet_copy_source_session.cc:205] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:19:59.777915 30656 ts_tablet_manager.cc:1875] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:19:59.781590 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(6575b94bd51546f1bd3bb862f012e7ca): 4581 bytes on disk
I20240501 08:19:59.782325 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(6575b94bd51546f1bd3bb862f012e7ca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240501 08:19:59.786700 30656 ts_tablet_manager.cc:1888] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:19:59.786981 30656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399d1a392b3f4d9e8531b6e0b5ad2a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:19:59.791052 30656 tablet_copy_client.cc:783] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Starting download of 5 data blocks...
I20240501 08:19:59.794531 30633 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:19:59.794962 30633 tablet_bootstrap.cc:492] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap complete.
I20240501 08:19:59.795279 30633 ts_tablet_manager.cc:1367] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.047s user 0.021s sys 0.002s
I20240501 08:19:59.796063 30633 raft_consensus.cc:348] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:19:59.796277 30633 raft_consensus.cc:729] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:19:59.796504 30633 consensus_queue.cc:260] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:19:59.797078 30633 ts_tablet_manager.cc:1398] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:19:59.797628 30656 tablet_copy_client.cc:647] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Starting download of 1 WAL segments...
I20240501 08:19:59.799321 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session cd3434d87b3c46c6848a711cdf6b150c-491674dfbf74498e8a787bebaa5e95da received from {username='slave'} at 127.25.107.2:46687
I20240501 08:19:59.799444 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session cd3434d87b3c46c6848a711cdf6b150c-491674dfbf74498e8a787bebaa5e95da on tablet 491674dfbf74498e8a787bebaa5e95da with peer cd3434d87b3c46c6848a711cdf6b150c
I20240501 08:19:59.800199 30656 tablet_copy_client.cc:523] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:19:59.804370 30656 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
W20240501 08:19:59.811122 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a25e1ddb987b4f8c8a4031053f581c7c: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:19:59.837108 30656 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:19:59.837553 30656 tablet_bootstrap.cc:492] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap complete.
I20240501 08:19:59.837857 30656 ts_tablet_manager.cc:1367] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.010s
I20240501 08:19:59.838724 30656 raft_consensus.cc:348] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:19:59.838968 30656 raft_consensus.cc:729] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:19:59.839541 30656 consensus_queue.cc:260] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:19:59.840498 30656 ts_tablet_manager.cc:1398] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 08:19:59.841274 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 97cdee0bf9d84645a1451f9822c5d699-399d1a392b3f4d9e8531b6e0b5ad2a61 received from {username='slave'} at 127.25.107.1:37175
I20240501 08:19:59.841400 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 97cdee0bf9d84645a1451f9822c5d699-399d1a392b3f4d9e8531b6e0b5ad2a61 on tablet 399d1a392b3f4d9e8531b6e0b5ad2a61 with peer 97cdee0bf9d84645a1451f9822c5d699
I20240501 08:19:59.885591 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(491674dfbf74498e8a787bebaa5e95da): 4921 bytes on disk
I20240501 08:19:59.886284 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(491674dfbf74498e8a787bebaa5e95da) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 08:19:59.941370 6768 raft_consensus.cc:1203] T 491674dfbf74498e8a787bebaa5e95da P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240501 08:19:59.944584 6572 connection.cc:535] client connection to 127.25.107.6:41219 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:19:59.944701 6703 connection.cc:535] client connection to 127.25.107.6:41219 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:19:59.945302 8893 connection.cc:535] client connection to 127.25.107.6:41219 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:19:59.945448 8893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
W20240501 08:19:59.946314 6963 connection.cc:535] client connection to 127.25.107.6:41219 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:19:59.946705 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(30b4f075057e42d0bb3f8cdabdfbc643): 4922 bytes on disk
I20240501 08:19:59.947356 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(30b4f075057e42d0bb3f8cdabdfbc643) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 08:19:59.948632 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(399d1a392b3f4d9e8531b6e0b5ad2a61): 4921 bytes on disk
W20240501 08:19:59.948773 29484 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:19:59.946525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.107.6:41219 (local address 127.25.107.2:43235)
0501 08:19:59.946755 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0501 08:19:59.946772 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 08:19:59.946876 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:19:59.948596 (+ 1720us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129}
W20240501 08:19:59.948873 6833 connection.cc:535] client connection to 127.25.107.6:41219 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:19:59.949234 30029 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:19:59.947294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.107.6:41219 (local address 127.25.107.4:44121)
0501 08:19:59.947452 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0501 08:19:59.947470 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0501 08:19:59.947580 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:19:59.949044 (+ 1464us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":41}
W20240501 08:19:59.949426 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:19:59.949537 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:19:59.949931 28151 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:19:59.945845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.107.6:41219 (local address 127.25.107.1:50641)
0501 08:19:59.946004 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0501 08:19:59.946024 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0501 08:19:59.946167 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:19:59.949793 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
W20240501 08:19:59.950080 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:19:59.950206 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:19:59.950317 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:19:59.950404 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:19:59.950485 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:19:59.951534 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(399d1a392b3f4d9e8531b6e0b5ad2a61) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240501 08:19:59.951856 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:19:59.951969 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:19:59.952046 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:19:59.954464 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:19:59.959900 8893 consensus_peers.cc:485] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:19:59.961620 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:19:59.968765 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:19:59.976612 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:19:59.981662 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:19:59.981669 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:19:59.991392 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:19:59.992754 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:19:59.993775 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:00.007280 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:00.070525 6637 raft_consensus.cc:1203] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13] Dedup: 3.13->[]
I20240501 08:20:00.148697 30408 raft_consensus.cc:998] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac: Attempting to remove follower 0a814bba1abe4653bcd418a83ec42370 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 0a814bba1abe4653bcd418a83ec42370 for more than 300 seconds (300.549s)
I20240501 08:20:00.149490 30408 consensus_queue.cc:237] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:00.156883 6768 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:20:00.157632 30732 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:20:00.178783 30732 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:00.185504 6768 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:00.188902 6497 catalog_manager.cc:5484] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 5 to 7, VOTER 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5) evicted. New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:00.212064 6896 consensus_queue.cc:237] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:00.214447 6768 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 08:20:00.214602 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7903106d7e74e1ea102b1abdbfce63e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:00.215162 30408 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:20:00.217862 30408 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:00.221319 6768 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:00.225731 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d7903106d7e74e1ea102b1abdbfce63e with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:00.228331 6492 catalog_manager.cc:5484] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 7 to 8, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:00.245471 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:20:00.262542 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:00.263890 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:20:00.265537 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:00.279053 6636 raft_consensus.cc:1203] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 08:20:00.284909 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:00.314162 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:00.380967 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:00.429674 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:00.443550 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:20:00.459748 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:00.465019 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:00.467336 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:00.474799 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:00.719141 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:20:00.733194 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:00.739778 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:00.782156 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:00.798669 30827 ts_tablet_manager.cc:902] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:20:00.818832 6877 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:00.855736 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d7903106d7e74e1ea102b1abdbfce63e: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:20:00.857578 30827 tablet_copy_client.cc:308] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:20:00.858532 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet d7903106d7e74e1ea102b1abdbfce63e from peer 19fdb76c8d1f41d7842e397adfc1ef3f ({username='slave'} at 127.25.107.8:32865)
I20240501 08:20:00.858723 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet d7903106d7e74e1ea102b1abdbfce63e from peer 19fdb76c8d1f41d7842e397adfc1ef3f at {username='slave'} at 127.25.107.8:32865: session id = 19fdb76c8d1f41d7842e397adfc1ef3f-d7903106d7e74e1ea102b1abdbfce63e
I20240501 08:20:00.862120 6919 tablet_copy_source_session.cc:205] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:00.864176 30827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7903106d7e74e1ea102b1abdbfce63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:00.871695 30827 tablet_copy_client.cc:783] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:00.880105 30827 tablet_copy_client.cc:647] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:00.881970 30827 tablet_copy_client.cc:523] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:00.885897 30827 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap starting.
I20240501 08:20:00.907673 6896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f"
dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37230
I20240501 08:20:00.907958 6896 raft_consensus.cc:595] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Received request to transfer leadership
I20240501 08:20:00.927224 30827 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:00.927722 30827 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f: Bootstrap complete.
I20240501 08:20:00.928047 30827 ts_tablet_manager.cc:1367] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent bootstrapping tablet: real 0.042s user 0.025s sys 0.000s
I20240501 08:20:00.928854 30827 raft_consensus.cc:348] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:00.929055 30827 raft_consensus.cc:729] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Initialized, Role: FOLLOWER
I20240501 08:20:00.929376 30827 consensus_queue.cc:260] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:00.930016 30827 ts_tablet_manager.cc:1398] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:20:00.930889 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-d7903106d7e74e1ea102b1abdbfce63e received from {username='slave'} at 127.25.107.8:32865
I20240501 08:20:00.931039 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 19fdb76c8d1f41d7842e397adfc1ef3f-d7903106d7e74e1ea102b1abdbfce63e on tablet d7903106d7e74e1ea102b1abdbfce63e with peer 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:20:00.934151 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:00.940588 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:00.941880 9005 maintenance_manager.cc:419] P 19fdb76c8d1f41d7842e397adfc1ef3f: Scheduling UndoDeltaBlockGCOp(d7903106d7e74e1ea102b1abdbfce63e): 4834 bytes on disk
I20240501 08:20:00.948405 8896 maintenance_manager.cc:643] P 19fdb76c8d1f41d7842e397adfc1ef3f: UndoDeltaBlockGCOp(d7903106d7e74e1ea102b1abdbfce63e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240501 08:20:01.148757 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:01.160226 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:20:01.162842 8938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:20:01.169332 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:01.188140 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:01.189973 30731 raft_consensus.cc:979] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: : Instructing follower 19fdb76c8d1f41d7842e397adfc1ef3f to start an election
I20240501 08:20:01.190128 30731 raft_consensus.cc:1067] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Signalling peer 19fdb76c8d1f41d7842e397adfc1ef3f to start an election
I20240501 08:20:01.191226 8958 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f"
dest_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f"
from {username='slave'} at 127.25.107.3:47407
I20240501 08:20:01.191452 8958 raft_consensus.cc:482] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:20:01.191542 8958 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:20:01.196051 8958 raft_consensus.cc:504] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:01.197587 8958 leader_election.cc:288] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 3 election: Requested vote from peers e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:20:01.198331 6896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "e3a1c4a5b44340be8a183d5d649b27ac"
I20240501 08:20:01.198573 6896 raft_consensus.cc:3032] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:20:01.198661 6896 raft_consensus.cc:729] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Running, Role: LEADER
I20240501 08:20:01.198983 6896 consensus_queue.cc:260] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:01.199429 6896 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:20:01.204209 6896 raft_consensus.cc:2454] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fdb76c8d1f41d7842e397adfc1ef3f in term 3.
I20240501 08:20:01.206732 8892 leader_election.cc:302] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f, e3a1c4a5b44340be8a183d5d649b27ac; no voters:
I20240501 08:20:01.207267 30416 raft_consensus.cc:2781] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:20:01.210732 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:20:01.210980 6637 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:20:01.214140 6637 raft_consensus.cc:2454] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fdb76c8d1f41d7842e397adfc1ef3f in term 3.
W20240501 08:20:01.226636 6571 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:20:01.242323 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:01.246592 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:01.249437 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:01.262713 30416 raft_consensus.cc:686] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 LEADER]: Becoming Leader. State: Replica: 19fdb76c8d1f41d7842e397adfc1ef3f, State: Running, Role: LEADER
I20240501 08:20:01.263208 30416 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:01.291281 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:01.293483 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:01.295187 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:01.333560 6491 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: term changed from 2 to 3, leader changed from e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) to 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8). New cstate: current_term: 3 leader_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
W20240501 08:20:01.346086 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:01.350049 30416 raft_consensus.cc:482] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:20:01.350297 30416 raft_consensus.cc:504] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:20:01.351259 30416 leader_election.cc:288] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:20:01.353832 8892 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:01.361493 8895 leader_election.cc:334] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:20:01.361698 8895 leader_election.cc:302] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fdb76c8d1f41d7842e397adfc1ef3f; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:20:01.361994 30416 raft_consensus.cc:2726] T 57ecbb30f7764fceb008baaa2c37da8c P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:20:01.384920 8958 raft_consensus.cc:1203] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20240501 08:20:01.391021 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:01.423287 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:01.437386 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:20:01.504989 6571 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:20:01.518505 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:01.529596 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:01.533841 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:01.560304 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:01.575510 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:01.666146 6962 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:20:01.682725 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:20:01.690961 6896 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Refusing update from remote peer 19fdb76c8d1f41d7842e397adfc1ef3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 08:20:01.691751 30416 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:20:01.721455 6637 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer 19fdb76c8d1f41d7842e397adfc1ef3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 08:20:01.722059 30904 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:20:01.740167 30627 raft_consensus.cc:998] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c: Attempting to remove follower a518ca5e9de24204bd028b0ce4a77d58 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a518ca5e9de24204bd028b0ce4a77d58 for more than 300 seconds (300.138s)
I20240501 08:20:01.741942 30627 consensus_queue.cc:237] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:20:01.744707 7027 raft_consensus.cc:1261] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:20:01.746127 30627 consensus_queue.cc:1035] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:20:01.765311 30980 raft_consensus.cc:2932] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:20:01.772251 6493 catalog_manager.cc:5484] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:20:01.773873 7027 raft_consensus.cc:2932] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:20:01.808005 6768 consensus_queue.cc:237] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:01.811131 7027 raft_consensus.cc:1261] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:20:01.811326 6702 consensus_peers.cc:485] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c -> Peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): Couldn't send request to peer e3a1c4a5b44340be8a183d5d649b27ac. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:01.812187 30976 consensus_queue.cc:1035] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:20:01.815269 30627 raft_consensus.cc:2932] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:01.816819 7027 raft_consensus.cc:2932] T 193de3c11ae54408a64ae3007bf88920 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:01.820168 6484 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 193de3c11ae54408a64ae3007bf88920 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:01.821931 6493 catalog_manager.cc:5484] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 6 to 7, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:01.843004 30980 raft_consensus.cc:998] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: Attempting to remove follower a518ca5e9de24204bd028b0ce4a77d58 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a518ca5e9de24204bd028b0ce4a77d58 for more than 300 seconds (300.207s)
I20240501 08:20:01.843789 30980 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:01.846091 6896 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:20:01.846110 8958 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:20:01.846727 30976 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:20:01.847064 30627 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:20:01.859805 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:01.867396 30627 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:01.867947 6896 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:01.869316 8957 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:01.875905 6493 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 6 to 7, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:20:01.883846 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a7a56c284444496b8579600a49f19d1d: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:01.928311 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:01.954624 30986 raft_consensus.cc:998] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699: Attempting to remove follower a518ca5e9de24204bd028b0ce4a77d58 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a518ca5e9de24204bd028b0ce4a77d58 for more than 300 seconds (300.199s)
I20240501 08:20:01.955399 30986 consensus_queue.cc:237] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
W20240501 08:20:01.961172 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:01.967211 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:20:01.980574 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:20:01.995204 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:01.997097 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:02.001505 6768 raft_consensus.cc:1261] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:20:02.003347 30857 consensus_queue.cc:1035] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:20:02.029294 30857 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:20:02.034795 6768 raft_consensus.cc:2932] T 29506d751571401f86bced7e6f5415d7 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:20:02.036679 6490 catalog_manager.cc:5484] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 6 to 7, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:20:02.050043 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:02.050983 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:02.051092 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:02.052763 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:02.064183 30986 raft_consensus.cc:998] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699: Attempting to remove follower a518ca5e9de24204bd028b0ce4a77d58 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a518ca5e9de24204bd028b0ce4a77d58 for more than 300 seconds (300.120s)
I20240501 08:20:02.065032 30986 consensus_queue.cc:237] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } }
I20240501 08:20:02.079908 30987 raft_consensus.cc:998] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699: Attempting to remove follower a518ca5e9de24204bd028b0ce4a77d58 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a518ca5e9de24204bd028b0ce4a77d58 for more than 300 seconds (300.402s)
I20240501 08:20:02.080513 30987 consensus_queue.cc:237] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:02.085562 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:02.097165 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:02.116230 6896 raft_consensus.cc:1261] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:20:02.116523 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:02.117348 8958 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:20:02.117412 6896 raft_consensus.cc:1261] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:20:02.118149 30857 consensus_queue.cc:1035] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:20:02.118690 30857 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:20:02.119145 30857 consensus_queue.cc:1035] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:20:02.127425 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:02.130856 30987 raft_consensus.cc:2932] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:02.139039 6896 raft_consensus.cc:2932] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:02.142469 6493 catalog_manager.cc:5484] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 5 to 6, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:02.149359 30986 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:20:02.155665 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:02.156591 6896 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:20:02.157207 6637 consensus_queue.cc:237] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:02.162750 6490 catalog_manager.cc:5484] T 13b4e2d5fe4846e8b3c6081abb3c53da P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
I20240501 08:20:02.163450 6898 raft_consensus.cc:1261] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Refusing update from remote peer 97cdee0bf9d84645a1451f9822c5d699: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:20:02.163770 6574 consensus_peers.cc:485] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 -> Peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): Couldn't send request to peer cd3434d87b3c46c6848a711cdf6b150c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78d1df7b0ea44f89b9f22c156f907c1e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:02.164193 30857 consensus_queue.cc:1035] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:20:02.167973 30857 raft_consensus.cc:2932] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.168140 6898 raft_consensus.cc:2932] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.161064 8958 raft_consensus.cc:2932] T 13b4e2d5fe4846e8b3c6081abb3c53da P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } }
W20240501 08:20:02.174862 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:20:02.177345 6493 catalog_manager.cc:5484] T 78d1df7b0ea44f89b9f22c156f907c1e P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 6 to 7, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New cstate: current_term: 1 leader_uuid: "97cdee0bf9d84645a1451f9822c5d699" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.178433 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 78d1df7b0ea44f89b9f22c156f907c1e with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:02.196543 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:02.209497 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:02.218194 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:02.238946 6963 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.8:41625, user_credentials={real_user=slave}} blocked reactor thread for 46870.9us
I20240501 08:20:02.244520 31050 raft_consensus.cc:998] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240: Attempting to remove follower a518ca5e9de24204bd028b0ce4a77d58 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a518ca5e9de24204bd028b0ce4a77d58 for more than 300 seconds (300.222s)
W20240501 08:20:02.244537 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:02.245647 31050 consensus_queue.cc:237] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:02.255266 8958 raft_consensus.cc:1261] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:20:02.255882 31050 consensus_queue.cc:1035] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:20:02.269281 8893 consensus_peers.cc:485] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:02.284976 30847 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:02.285653 8958 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:02.291357 6493 catalog_manager.cc:5484] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:20:02.304785 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:02.307468 7028 consensus_queue.cc:237] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:02.311239 8958 raft_consensus.cc:1261] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:20:02.312662 30847 consensus_queue.cc:1035] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:20:02.315663 31050 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.316454 8958 raft_consensus.cc:2932] T e9629fbbb4554edc9ba341d42b51f252 P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
W20240501 08:20:02.320695 6965 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): Couldn't send request to peer cd3434d87b3c46c6848a711cdf6b150c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9629fbbb4554edc9ba341d42b51f252. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:02.321012 6483 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e9629fbbb4554edc9ba341d42b51f252 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:02.322427 6490 catalog_manager.cc:5484] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
W20240501 08:20:02.332336 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:02.350314 6898 consensus_queue.cc:237] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:02.352545 8958 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:20:02.353143 30408 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:20:02.354884 31049 raft_consensus.cc:998] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240: Attempting to remove follower a518ca5e9de24204bd028b0ce4a77d58 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a518ca5e9de24204bd028b0ce4a77d58 for more than 300 seconds (300.409s)
I20240501 08:20:02.355589 31049 consensus_queue.cc:237] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
W20240501 08:20:02.357913 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:02.359354 6637 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:20:02.359112 31081 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:02.359860 8957 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:02.360394 31049 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:20:02.361033 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:02.365868 6493 catalog_manager.cc:5484] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:02.366408 30847 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:02.370579 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet d7903106d7e74e1ea102b1abdbfce63e with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40900
I20240501 08:20:02.371259 31085 tablet_replica.cc:330] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:20:02.371548 31085 raft_consensus.cc:2229] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:20:02.371963 31085 raft_consensus.cc:2258] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:20:02.372366 6637 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:02.373217 31085 ts_tablet_manager.cc:1875] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:20:02.380124 6493 catalog_manager.cc:5484] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 7 to 8, VOTER a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7) evicted. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
W20240501 08:20:02.384521 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:02.386200 6898 consensus_queue.cc:237] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
W20240501 08:20:02.389197 6835 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): Couldn't send request to peer cd3434d87b3c46c6848a711cdf6b150c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:02.390946 7028 consensus_queue.cc:237] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:02.391892 31089 ts_tablet_manager.cc:902] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:20:02.393472 8957 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240501 08:20:02.394627 6965 consensus_peers.cc:485] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): Couldn't send request to peer cd3434d87b3c46c6848a711cdf6b150c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:02.395169 6637 raft_consensus.cc:1261] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:20:02.395846 31049 consensus_queue.cc:1035] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:20:02.398496 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:02.398608 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:02.399483 31049 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.400017 31083 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:20:02.403148 6637 raft_consensus.cc:2932] T 45d076303ea24d7fa0fed5a9d132019c P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.403306 31083 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.407596 8957 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.408252 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d7903106d7e74e1ea102b1abdbfce63e with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:02.410084 6483 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 45d076303ea24d7fa0fed5a9d132019c with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:02.410161 6493 catalog_manager.cc:5484] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 9 to 10, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.414515 6490 catalog_manager.cc:5484] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: config changed from index 8 to 9, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New cstate: current_term: 1 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.436028 31089 tablet_copy_client.cc:308] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:20:02.438175 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet 193de3c11ae54408a64ae3007bf88920 from peer e3a1c4a5b44340be8a183d5d649b27ac ({username='slave'} at 127.25.107.3:42535)
I20240501 08:20:02.438392 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet 193de3c11ae54408a64ae3007bf88920 from peer e3a1c4a5b44340be8a183d5d649b27ac at {username='slave'} at 127.25.107.3:42535: session id = e3a1c4a5b44340be8a183d5d649b27ac-193de3c11ae54408a64ae3007bf88920
I20240501 08:20:02.441704 6789 tablet_copy_source_session.cc:205] T 193de3c11ae54408a64ae3007bf88920 P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:02.443156 31089 ts_tablet_manager.cc:1875] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:20:02.447582 31085 ts_tablet_manager.cc:1888] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 08:20:02.447865 31085 log.cc:1196] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/d7903106d7e74e1ea102b1abdbfce63e
I20240501 08:20:02.449401 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet d7903106d7e74e1ea102b1abdbfce63e (table rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]) successfully deleted
I20240501 08:20:02.451056 31089 ts_tablet_manager.cc:1888] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 08:20:02.451309 31089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193de3c11ae54408a64ae3007bf88920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:02.454715 31089 tablet_copy_client.cc:783] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:02.456467 6898 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:02.458441 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 08:20:02.458992 31083 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:20:02.461529 31081 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:02.462098 6637 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:02.462754 31089 tablet_copy_client.cc:647] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:02.464668 31089 tablet_copy_client.cc:523] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:02.467497 6493 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 13 to 14, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:02.469828 31089 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
I20240501 08:20:02.472916 6898 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:02.474138 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:40900
I20240501 08:20:02.475661 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 08:20:02.475679 6768 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 08:20:02.476511 31083 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 08:20:02.476933 31085 tablet_replica.cc:330] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:20:02.477057 30408 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:20:02.477173 31085 raft_consensus.cc:2229] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:20:02.477458 31085 raft_consensus.cc:2258] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:20:02.478379 31085 ts_tablet_manager.cc:1875] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:20:02.479171 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): Couldn't send request to peer cd3434d87b3c46c6848a711cdf6b150c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:02.480872 31081 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.485425 6637 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.487188 31085 ts_tablet_manager.cc:1888] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240501 08:20:02.487385 31085 log.cc:1196] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:20:02.488307 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 30b4f075057e42d0bb3f8cdabdfbc643 (table rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]) successfully deleted
I20240501 08:20:02.494174 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with cas_config_opid_index 14: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:02.494764 6491 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 14 to 15, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:02.506917 31089 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:02.507349 31089 tablet_bootstrap.cc:492] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap complete.
I20240501 08:20:02.507651 31089 ts_tablet_manager.cc:1367] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.038s user 0.020s sys 0.005s
I20240501 08:20:02.508409 31089 raft_consensus.cc:348] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:02.508630 31089 raft_consensus.cc:729] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:20:02.508883 31089 consensus_queue.cc:260] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:02.509529 31089 ts_tablet_manager.cc:1398] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:20:02.510499 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-193de3c11ae54408a64ae3007bf88920 received from {username='slave'} at 127.25.107.3:42535
I20240501 08:20:02.510650 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-193de3c11ae54408a64ae3007bf88920 on tablet 193de3c11ae54408a64ae3007bf88920 with peer e3a1c4a5b44340be8a183d5d649b27ac
W20240501 08:20:02.574157 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:02.601099 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 87 similar messages]
I20240501 08:20:02.607349 31138 ts_tablet_manager.cc:902] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c: Initiating tablet copy from peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:20:02.622727 31138 tablet_copy_client.cc:308] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.1:45193
I20240501 08:20:02.623479 6658 tablet_copy_service.cc:140] P 97cdee0bf9d84645a1451f9822c5d699: Received BeginTabletCopySession request for tablet 78d1df7b0ea44f89b9f22c156f907c1e from peer cd3434d87b3c46c6848a711cdf6b150c ({username='slave'} at 127.25.107.2:46687)
I20240501 08:20:02.623611 6658 tablet_copy_service.cc:161] P 97cdee0bf9d84645a1451f9822c5d699: Beginning new tablet copy session on tablet 78d1df7b0ea44f89b9f22c156f907c1e from peer cd3434d87b3c46c6848a711cdf6b150c at {username='slave'} at 127.25.107.2:46687: session id = cd3434d87b3c46c6848a711cdf6b150c-78d1df7b0ea44f89b9f22c156f907c1e
I20240501 08:20:02.627727 6658 tablet_copy_source_session.cc:205] T 78d1df7b0ea44f89b9f22c156f907c1e P 97cdee0bf9d84645a1451f9822c5d699: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:02.628706 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(193de3c11ae54408a64ae3007bf88920): 4582 bytes on disk
I20240501 08:20:02.629470 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(193de3c11ae54408a64ae3007bf88920) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240501 08:20:02.630132 31138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d1df7b0ea44f89b9f22c156f907c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:02.639194 31138 tablet_copy_client.cc:783] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:02.658814 31138 tablet_copy_client.cc:647] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:02.661545 31138 tablet_copy_client.cc:523] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:02.664964 31138 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
W20240501 08:20:02.712301 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:20:02.723454 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:02.725463 31138 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:02.726218 31138 tablet_bootstrap.cc:492] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap complete.
I20240501 08:20:02.728094 31138 ts_tablet_manager.cc:1367] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.063s user 0.027s sys 0.000s
I20240501 08:20:02.729373 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:02.728984 31138 raft_consensus.cc:348] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:02.733742 31138 raft_consensus.cc:729] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:20:02.734161 31138 consensus_queue.cc:260] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:02.747999 31138 ts_tablet_manager.cc:1398] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240501 08:20:02.752943 6658 tablet_copy_service.cc:342] P 97cdee0bf9d84645a1451f9822c5d699: Request end of tablet copy session cd3434d87b3c46c6848a711cdf6b150c-78d1df7b0ea44f89b9f22c156f907c1e received from {username='slave'} at 127.25.107.2:46687
I20240501 08:20:02.753088 6658 tablet_copy_service.cc:433] P 97cdee0bf9d84645a1451f9822c5d699: ending tablet copy session cd3434d87b3c46c6848a711cdf6b150c-78d1df7b0ea44f89b9f22c156f907c1e on tablet 78d1df7b0ea44f89b9f22c156f907c1e with peer cd3434d87b3c46c6848a711cdf6b150c
I20240501 08:20:02.763649 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:02.766111 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:02.769588 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:02.783599 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:02.794579 31138 ts_tablet_manager.cc:902] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:20:02.836135 31138 tablet_copy_client.cc:308] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:20:02.836967 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet e9629fbbb4554edc9ba341d42b51f252 from peer cd3434d87b3c46c6848a711cdf6b150c ({username='slave'} at 127.25.107.2:54393)
I20240501 08:20:02.837210 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet e9629fbbb4554edc9ba341d42b51f252 from peer cd3434d87b3c46c6848a711cdf6b150c at {username='slave'} at 127.25.107.2:54393: session id = cd3434d87b3c46c6848a711cdf6b150c-e9629fbbb4554edc9ba341d42b51f252
I20240501 08:20:02.841253 7049 tablet_copy_source_session.cc:205] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:02.846928 31138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9629fbbb4554edc9ba341d42b51f252. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:02.854434 31138 tablet_copy_client.cc:783] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:02.861264 31138 tablet_copy_client.cc:647] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:02.863092 31138 tablet_copy_client.cc:523] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:02.866847 31138 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:20:02.877298 6898 raft_consensus.cc:1203] T 193de3c11ae54408a64ae3007bf88920 P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:20:02.899600 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(78d1df7b0ea44f89b9f22c156f907c1e): 4582 bytes on disk
I20240501 08:20:02.900354 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(78d1df7b0ea44f89b9f22c156f907c1e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 08:20:02.902352 31183 ts_tablet_manager.cc:902] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:20:02.935869 31138 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:02.937491 31138 tablet_bootstrap.cc:492] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap complete.
W20240501 08:20:02.953337 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 193de3c11ae54408a64ae3007bf88920: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:02.956593 31138 ts_tablet_manager.cc:1367] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.090s user 0.027s sys 0.002s
I20240501 08:20:02.959187 31138 raft_consensus.cc:348] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:02.959460 31138 raft_consensus.cc:729] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:20:02.960376 31196 ts_tablet_manager.cc:902] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Initiating tablet copy from peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:20:02.961130 31183 tablet_copy_client.cc:308] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:20:02.961841 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer cd3434d87b3c46c6848a711cdf6b150c ({username='slave'} at 127.25.107.2:34709)
I20240501 08:20:02.962013 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer cd3434d87b3c46c6848a711cdf6b150c at {username='slave'} at 127.25.107.2:34709: session id = cd3434d87b3c46c6848a711cdf6b150c-30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:20:02.965399 6919 tablet_copy_source_session.cc:205] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:02.967229 31183 ts_tablet_manager.cc:1875] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:20:02.959731 31138 consensus_queue.cc:260] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
W20240501 08:20:02.971113 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:02.971313 31138 ts_tablet_manager.cc:1398] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240501 08:20:02.972555 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session cd3434d87b3c46c6848a711cdf6b150c-e9629fbbb4554edc9ba341d42b51f252 received from {username='slave'} at 127.25.107.2:54393
I20240501 08:20:02.972680 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session cd3434d87b3c46c6848a711cdf6b150c-e9629fbbb4554edc9ba341d42b51f252 on tablet e9629fbbb4554edc9ba341d42b51f252 with peer cd3434d87b3c46c6848a711cdf6b150c
I20240501 08:20:02.971414 31203 ts_tablet_manager.cc:902] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:20:02.984004 31196 tablet_copy_client.cc:308] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.4:43623
I20240501 08:20:02.984714 7049 tablet_copy_service.cc:140] P 28b7613bd4834f3ab245c82dd7e80240: Received BeginTabletCopySession request for tablet 45d076303ea24d7fa0fed5a9d132019c from peer cd3434d87b3c46c6848a711cdf6b150c ({username='slave'} at 127.25.107.2:54393)
I20240501 08:20:02.984889 7049 tablet_copy_service.cc:161] P 28b7613bd4834f3ab245c82dd7e80240: Beginning new tablet copy session on tablet 45d076303ea24d7fa0fed5a9d132019c from peer cd3434d87b3c46c6848a711cdf6b150c at {username='slave'} at 127.25.107.2:54393: session id = cd3434d87b3c46c6848a711cdf6b150c-45d076303ea24d7fa0fed5a9d132019c
I20240501 08:20:02.989002 7049 tablet_copy_source_session.cc:205] T 45d076303ea24d7fa0fed5a9d132019c P 28b7613bd4834f3ab245c82dd7e80240: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:02.990676 31196 ts_tablet_manager.cc:1875] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:20:03.003803 31203 tablet_copy_client.cc:308] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:20:03.005448 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet d7903106d7e74e1ea102b1abdbfce63e from peer cd3434d87b3c46c6848a711cdf6b150c ({username='slave'} at 127.25.107.2:34709)
I20240501 08:20:03.005630 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet d7903106d7e74e1ea102b1abdbfce63e from peer cd3434d87b3c46c6848a711cdf6b150c at {username='slave'} at 127.25.107.2:34709: session id = cd3434d87b3c46c6848a711cdf6b150c-d7903106d7e74e1ea102b1abdbfce63e
I20240501 08:20:03.011302 6919 tablet_copy_source_session.cc:205] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:03.017625 31203 ts_tablet_manager.cc:1875] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:20:03.020910 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:03.025413 31203 ts_tablet_manager.cc:1888] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.8
I20240501 08:20:03.025684 31203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7903106d7e74e1ea102b1abdbfce63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:03.029317 31203 tablet_copy_client.cc:783] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:03.035841 31183 ts_tablet_manager.cc:1888] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.13
I20240501 08:20:03.038090 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4f075057e42d0bb3f8cdabdfbc643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:03.040220 31203 tablet_copy_client.cc:647] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:03.043262 31203 tablet_copy_client.cc:523] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:03.049517 31196 ts_tablet_manager.cc:1888] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 08:20:03.049805 31196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d076303ea24d7fa0fed5a9d132019c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:03.052911 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:03.059507 31183 tablet_copy_client.cc:783] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:03.059772 31203 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:20:03.068905 31183 tablet_copy_client.cc:647] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:03.070081 31196 tablet_copy_client.cc:783] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:03.071030 31183 tablet_copy_client.cc:523] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:03.075225 31183 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:20:03.091224 31196 tablet_copy_client.cc:647] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:03.094739 31196 tablet_copy_client.cc:523] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:03.100342 31196 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:20:03.104974 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:03.112707 31203 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:03.122721 31203 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap complete.
I20240501 08:20:03.123103 31203 ts_tablet_manager.cc:1367] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.007s
I20240501 08:20:03.124075 31203 raft_consensus.cc:348] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:03.124389 31203 raft_consensus.cc:729] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:20:03.124689 31203 consensus_queue.cc:260] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:03.125324 31203 ts_tablet_manager.cc:1398] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:20:03.126041 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session cd3434d87b3c46c6848a711cdf6b150c-d7903106d7e74e1ea102b1abdbfce63e received from {username='slave'} at 127.25.107.2:34709
I20240501 08:20:03.126165 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session cd3434d87b3c46c6848a711cdf6b150c-d7903106d7e74e1ea102b1abdbfce63e on tablet d7903106d7e74e1ea102b1abdbfce63e with peer cd3434d87b3c46c6848a711cdf6b150c
I20240501 08:20:03.153224 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(d7903106d7e74e1ea102b1abdbfce63e): 4834 bytes on disk
I20240501 08:20:03.154038 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(d7903106d7e74e1ea102b1abdbfce63e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 08:20:03.155470 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(e9629fbbb4554edc9ba341d42b51f252): 4581 bytes on disk
I20240501 08:20:03.156203 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(e9629fbbb4554edc9ba341d42b51f252) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240501 08:20:03.180447 31196 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:03.180904 31196 tablet_bootstrap.cc:492] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap complete.
I20240501 08:20:03.181936 31196 ts_tablet_manager.cc:1367] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.082s user 0.023s sys 0.016s
W20240501 08:20:03.182893 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:03.182876 31196 raft_consensus.cc:348] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:03.183111 31196 raft_consensus.cc:729] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:20:03.183359 31196 consensus_queue.cc:260] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:03.184746 31196 ts_tablet_manager.cc:1398] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:20:03.187964 7049 tablet_copy_service.cc:342] P 28b7613bd4834f3ab245c82dd7e80240: Request end of tablet copy session cd3434d87b3c46c6848a711cdf6b150c-45d076303ea24d7fa0fed5a9d132019c received from {username='slave'} at 127.25.107.2:54393
I20240501 08:20:03.188122 7049 tablet_copy_service.cc:433] P 28b7613bd4834f3ab245c82dd7e80240: ending tablet copy session cd3434d87b3c46c6848a711cdf6b150c-45d076303ea24d7fa0fed5a9d132019c on tablet 45d076303ea24d7fa0fed5a9d132019c with peer cd3434d87b3c46c6848a711cdf6b150c
I20240501 08:20:03.188174 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:03.203755 6768 raft_consensus.cc:1203] T 78d1df7b0ea44f89b9f22c156f907c1e P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:20:03.205830 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:03.213239 31183 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:03.213793 31183 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap complete.
I20240501 08:20:03.214570 31183 ts_tablet_manager.cc:1367] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.139s user 0.027s sys 0.003s
I20240501 08:20:03.216105 31183 raft_consensus.cc:348] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:03.220556 31183 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:20:03.220955 31183 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:03.222532 31183 ts_tablet_manager.cc:1398] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 08:20:03.224318 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session cd3434d87b3c46c6848a711cdf6b150c-30b4f075057e42d0bb3f8cdabdfbc643 received from {username='slave'} at 127.25.107.2:34709
I20240501 08:20:03.224455 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session cd3434d87b3c46c6848a711cdf6b150c-30b4f075057e42d0bb3f8cdabdfbc643 on tablet 30b4f075057e42d0bb3f8cdabdfbc643 with peer cd3434d87b3c46c6848a711cdf6b150c
W20240501 08:20:03.300021 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240501 08:20:03.320923 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:03.324977 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:03.325444 6768 raft_consensus.cc:1203] T 45d076303ea24d7fa0fed5a9d132019c P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 08:20:03.343251 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:03.345250 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:03.348315 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:03.353013 6486 maintenance_manager.cc:419] P d978dc7272e947d9938cb39d0d6c1eb8: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7467 bytes on disk
W20240501 08:20:03.363713 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240501 08:20:03.386099 6485 maintenance_manager.cc:643] P d978dc7272e947d9938cb39d0d6c1eb8: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.033s user 0.012s sys 0.003s Metrics: {"dirs.queue_time_us":16720,"thread_start_us":21225,"threads_started":1}
I20240501 08:20:03.389878 6768 raft_consensus.cc:1203] T e9629fbbb4554edc9ba341d42b51f252 P cd3434d87b3c46c6848a711cdf6b150c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 08:20:03.408985 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(30b4f075057e42d0bb3f8cdabdfbc643): 4922 bytes on disk
I20240501 08:20:03.409713 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(30b4f075057e42d0bb3f8cdabdfbc643) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240501 08:20:03.410248 6768 raft_consensus.cc:1203] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.13->[3.14-3.15] Dedup: 3.15->[]
I20240501 08:20:03.412256 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(45d076303ea24d7fa0fed5a9d132019c): 4834 bytes on disk
I20240501 08:20:03.414582 6768 raft_consensus.cc:1203] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 08:20:03.415544 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(45d076303ea24d7fa0fed5a9d132019c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240501 08:20:03.528409 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:03.531833 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:03.608546 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:03.611414 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:03.627005 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:03.655668 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:03.669615 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:03.728164 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:03.808921 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:03.815603 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:03.833722 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:03.833772 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:03.837766 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:03.886396 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:03.914305 8957 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:03.921460 6637 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer 19fdb76c8d1f41d7842e397adfc1ef3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 08:20:03.922180 30904 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:20:03.929754 6637 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:03.928973 31340 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:03.941985 6490 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 10 to 12, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 3 leader_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:03.949749 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f090fda814f455a88dccd6680bdd05f with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:37122
I20240501 08:20:03.951455 31349 tablet_replica.cc:330] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:20:03.951675 31349 raft_consensus.cc:2229] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:20:03.951869 31349 raft_consensus.cc:2258] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:20:03.952955 31349 ts_tablet_manager.cc:1875] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:20:03.958384 8957 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
W20240501 08:20:03.960014 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:20:03.960996 6637 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer 19fdb76c8d1f41d7842e397adfc1ef3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 08:20:03.961876 30904 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:20:03.964871 30904 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:20:03.966836 6637 raft_consensus.cc:2932] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:20:03.969764 31349 ts_tablet_manager.cc:1888] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240501 08:20:03.969980 31349 log.cc:1196] T 5f090fda814f455a88dccd6680bdd05f P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/5f090fda814f455a88dccd6680bdd05f
I20240501 08:20:03.970885 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet 5f090fda814f455a88dccd6680bdd05f (table rebalance_test_table_0 [id=314bb13099e14e298e69260ae58eb62c]) successfully deleted
I20240501 08:20:03.972340 6482 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5f090fda814f455a88dccd6680bdd05f with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240501 08:20:03.974133 8894 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f -> Peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): Couldn't send request to peer 28b7613bd4834f3ab245c82dd7e80240. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:03.977164 6496 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 12 to 13, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New cstate: current_term: 3 leader_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
W20240501 08:20:04.005578 6481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240501 08:20:04.012813 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 193de3c11ae54408a64ae3007bf88920: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:04.033859 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:04.034068 6896 consensus_queue.cc:237] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:04.036087 8957 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:20:04.036644 31250 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:20:04.039317 31081 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:04.040499 8957 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:04.046402 6496 catalog_manager.cc:5484] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 10 to 11, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:04.051044 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet d7903106d7e74e1ea102b1abdbfce63e with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:40900
I20240501 08:20:04.051729 31382 tablet_replica.cc:330] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:20:04.052021 31382 raft_consensus.cc:2229] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:20:04.052197 31382 raft_consensus.cc:2258] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:20:04.052547 6896 consensus_queue.cc:237] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:20:04.052999 31382 ts_tablet_manager.cc:1875] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:20:04.054704 8957 raft_consensus.cc:1261] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240501 08:20:04.054795 6834 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623): Couldn't send request to peer 28b7613bd4834f3ab245c82dd7e80240. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:04.055325 31250 consensus_queue.cc:1035] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:20:04.058774 31250 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:20:04.063208 8957 raft_consensus.cc:2932] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:20:04.065235 31382 ts_tablet_manager.cc:1888] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 08:20:04.065371 31382 log.cc:1196] T d7903106d7e74e1ea102b1abdbfce63e P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/d7903106d7e74e1ea102b1abdbfce63e
I20240501 08:20:04.066118 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet d7903106d7e74e1ea102b1abdbfce63e (table rebalance_test_table_8 [id=a24f4747208b4f7e9f25ac196652ebd4]) successfully deleted
I20240501 08:20:04.067410 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d7903106d7e74e1ea102b1abdbfce63e with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:04.069489 6496 catalog_manager.cc:5484] T d7903106d7e74e1ea102b1abdbfce63e P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 11 to 12, VOTER 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4) added. New cstate: current_term: 2 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:20:04.139910 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:04.155237 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:04.161558 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:04.174417 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:04.180403 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:04.285537 6769 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:04.290876 8957 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:20:04.291517 30980 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:20:04.295768 31248 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:04.296352 8957 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:04.302521 6493 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f reported cstate change: config changed from index 7 to 8, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:04.308620 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet a7a56c284444496b8579600a49f19d1d with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:37122
I20240501 08:20:04.308981 31349 tablet_replica.cc:330] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:20:04.309190 31349 raft_consensus.cc:2229] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:20:04.309423 31349 raft_consensus.cc:2258] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:20:04.310284 31349 ts_tablet_manager.cc:1875] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:20:04.310350 6769 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
W20240501 08:20:04.312548 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): Couldn't send request to peer e3a1c4a5b44340be8a183d5d649b27ac. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:04.314471 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:04.320019 31349 ts_tablet_manager.cc:1888] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 08:20:04.322405 8957 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 08:20:04.322743 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:20:04.323110 31248 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:20:04.325876 31349 log.cc:1196] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7a56c284444496b8579600a49f19d1d
I20240501 08:20:04.325851 30980 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:04.326529 8957 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:04.329226 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet a7a56c284444496b8579600a49f19d1d (table rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]) successfully deleted
I20240501 08:20:04.331285 6484 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a7a56c284444496b8579600a49f19d1d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:04.332015 6491 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 8 to 9, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
W20240501 08:20:04.417838 8893 consensus_peers.cc:485] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:04.425988 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:20:04.445156 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:04.445778 31476 ts_tablet_manager.cc:902] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Initiating tablet copy from peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625)
I20240501 08:20:04.459338 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:04.507197 31486 ts_tablet_manager.cc:902] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
I20240501 08:20:04.512817 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:04.521077 31476 tablet_copy_client.cc:308] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.8:41625
I20240501 08:20:04.521931 31486 tablet_copy_client.cc:308] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:20:04.522248 8979 tablet_copy_service.cc:140] P 19fdb76c8d1f41d7842e397adfc1ef3f: Received BeginTabletCopySession request for tablet 5f090fda814f455a88dccd6680bdd05f from peer 28b7613bd4834f3ab245c82dd7e80240 ({username='slave'} at 127.25.107.4:59601)
I20240501 08:20:04.522457 8979 tablet_copy_service.cc:161] P 19fdb76c8d1f41d7842e397adfc1ef3f: Beginning new tablet copy session on tablet 5f090fda814f455a88dccd6680bdd05f from peer 28b7613bd4834f3ab245c82dd7e80240 at {username='slave'} at 127.25.107.4:59601: session id = 28b7613bd4834f3ab245c82dd7e80240-5f090fda814f455a88dccd6680bdd05f
I20240501 08:20:04.522534 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:04.526149 8979 tablet_copy_source_session.cc:205] T 5f090fda814f455a88dccd6680bdd05f P 19fdb76c8d1f41d7842e397adfc1ef3f: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:04.526791 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet d7903106d7e74e1ea102b1abdbfce63e from peer 28b7613bd4834f3ab245c82dd7e80240 ({username='slave'} at 127.25.107.4:38861)
I20240501 08:20:04.526954 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet d7903106d7e74e1ea102b1abdbfce63e from peer 28b7613bd4834f3ab245c82dd7e80240 at {username='slave'} at 127.25.107.4:38861: session id = 28b7613bd4834f3ab245c82dd7e80240-d7903106d7e74e1ea102b1abdbfce63e
I20240501 08:20:04.530995 31476 ts_tablet_manager.cc:1875] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:20:04.535120 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:04.538326 6919 tablet_copy_source_session.cc:205] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:04.542273 31486 ts_tablet_manager.cc:1875] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:20:04.543686 31476 ts_tablet_manager.cc:1888] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.7
I20240501 08:20:04.543939 31476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f090fda814f455a88dccd6680bdd05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:04.551240 31476 tablet_copy_client.cc:783] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Starting download of 5 data blocks...
W20240501 08:20:04.555164 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:04.556717 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:04.563206 31476 tablet_copy_client.cc:647] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Starting download of 1 WAL segments...
W20240501 08:20:04.565997 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:04.566622 31486 ts_tablet_manager.cc:1888] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 08:20:04.566946 31486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7903106d7e74e1ea102b1abdbfce63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:04.567087 31476 tablet_copy_client.cc:523] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:04.571262 31476 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:20:04.587626 31486 tablet_copy_client.cc:783] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:04.594198 31486 tablet_copy_client.cc:647] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:04.597436 31486 tablet_copy_client.cc:523] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:04.601668 31486 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap starting.
I20240501 08:20:04.621436 31476 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:04.621981 31476 tablet_bootstrap.cc:492] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap complete.
I20240501 08:20:04.622357 31476 ts_tablet_manager.cc:1367] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.051s user 0.022s sys 0.008s
I20240501 08:20:04.623121 31476 raft_consensus.cc:348] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:20:04.623384 31476 raft_consensus.cc:729] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:20:04.623685 31476 consensus_queue.cc:260] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:20:04.624403 31476 ts_tablet_manager.cc:1398] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:20:04.625316 8979 tablet_copy_service.cc:342] P 19fdb76c8d1f41d7842e397adfc1ef3f: Request end of tablet copy session 28b7613bd4834f3ab245c82dd7e80240-5f090fda814f455a88dccd6680bdd05f received from {username='slave'} at 127.25.107.4:59601
I20240501 08:20:04.625455 8979 tablet_copy_service.cc:433] P 19fdb76c8d1f41d7842e397adfc1ef3f: ending tablet copy session 28b7613bd4834f3ab245c82dd7e80240-5f090fda814f455a88dccd6680bdd05f on tablet 5f090fda814f455a88dccd6680bdd05f with peer 28b7613bd4834f3ab245c82dd7e80240
W20240501 08:20:04.640122 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:04.656998 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:04.657109 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:04.664469 31486 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:04.665071 31486 tablet_bootstrap.cc:492] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Bootstrap complete.
I20240501 08:20:04.665488 31486 ts_tablet_manager.cc:1367] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Time spent bootstrapping tablet: real 0.064s user 0.027s sys 0.002s
I20240501 08:20:04.666385 31486 raft_consensus.cc:348] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:20:04.666702 31486 raft_consensus.cc:729] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Initialized, Role: FOLLOWER
I20240501 08:20:04.667029 31486 consensus_queue.cc:260] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:20:04.667764 31486 ts_tablet_manager.cc:1398] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:20:04.668566 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session 28b7613bd4834f3ab245c82dd7e80240-d7903106d7e74e1ea102b1abdbfce63e received from {username='slave'} at 127.25.107.4:38861
I20240501 08:20:04.668709 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session 28b7613bd4834f3ab245c82dd7e80240-d7903106d7e74e1ea102b1abdbfce63e on tablet d7903106d7e74e1ea102b1abdbfce63e with peer 28b7613bd4834f3ab245c82dd7e80240
W20240501 08:20:04.682698 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 08:20:04.687533 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:04.690320 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:04.721765 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:04.731539 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:04.737468 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:04.757048 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:04.758436 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:04.797319 31511 ts_tablet_manager.cc:902] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
W20240501 08:20:04.802008 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:04.809178 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(5f090fda814f455a88dccd6680bdd05f): 4834 bytes on disk
I20240501 08:20:04.809912 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(5f090fda814f455a88dccd6680bdd05f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 08:20:04.811247 7075 maintenance_manager.cc:419] P 28b7613bd4834f3ab245c82dd7e80240: Scheduling UndoDeltaBlockGCOp(d7903106d7e74e1ea102b1abdbfce63e): 4834 bytes on disk
I20240501 08:20:04.811872 6966 maintenance_manager.cc:643] P 28b7613bd4834f3ab245c82dd7e80240: UndoDeltaBlockGCOp(d7903106d7e74e1ea102b1abdbfce63e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240501 08:20:04.826520 31511 tablet_copy_client.cc:308] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:20:04.827224 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet a7a56c284444496b8579600a49f19d1d from peer e3a1c4a5b44340be8a183d5d649b27ac ({username='slave'} at 127.25.107.3:42535)
I20240501 08:20:04.827419 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet a7a56c284444496b8579600a49f19d1d from peer e3a1c4a5b44340be8a183d5d649b27ac at {username='slave'} at 127.25.107.3:42535: session id = e3a1c4a5b44340be8a183d5d649b27ac-a7a56c284444496b8579600a49f19d1d
I20240501 08:20:04.831390 6789 tablet_copy_source_session.cc:205] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:04.833431 31511 ts_tablet_manager.cc:1875] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240501 08:20:04.839752 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240501 08:20:04.846149 31511 ts_tablet_manager.cc:1888] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240501 08:20:04.846504 31511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a56c284444496b8579600a49f19d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:04.852847 31511 tablet_copy_client.cc:783] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 5 data blocks...
W20240501 08:20:04.856999 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:04.862358 31511 tablet_copy_client.cc:647] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:04.864313 31511 tablet_copy_client.cc:523] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:04.871244 31511 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
W20240501 08:20:04.902638 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:04.921310 7028 raft_consensus.cc:1203] T d7903106d7e74e1ea102b1abdbfce63e P 28b7613bd4834f3ab245c82dd7e80240 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 08:20:04.922827 31511 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:04.923494 31511 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap complete.
I20240501 08:20:04.923856 31511 ts_tablet_manager.cc:1367] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.053s user 0.030s sys 0.009s
I20240501 08:20:04.924945 31511 raft_consensus.cc:348] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:04.925302 31511 raft_consensus.cc:729] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:20:04.925597 31511 consensus_queue.cc:260] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:04.926290 31511 ts_tablet_manager.cc:1398] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:20:04.928607 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-a7a56c284444496b8579600a49f19d1d received from {username='slave'} at 127.25.107.3:42535
I20240501 08:20:04.928759 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-a7a56c284444496b8579600a49f19d1d on tablet a7a56c284444496b8579600a49f19d1d with peer e3a1c4a5b44340be8a183d5d649b27ac
I20240501 08:20:04.944767 6896 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:04.947005 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20240501 08:20:04.947657 31389 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 08:20:04.950814 31534 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:04.951531 6637 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
W20240501 08:20:04.957255 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:04.957641 6497 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 15 to 16, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
W20240501 08:20:04.989527 8893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240501 08:20:05.010576 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:40900
I20240501 08:20:05.012382 6896 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:05.014532 6768 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20240501 08:20:05.014868 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20240501 08:20:05.015331 31389 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 08:20:05.015795 31534 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240501 08:20:05.017099 6768 raft_consensus.cc:858] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }.
Pending config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }:
New config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
W20240501 08:20:05.017232 6768 raft_consensus.cc:1565] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 NON_PARTICIPANT]: Could not prepare op 'term: 3 index: 17' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }.
Pending config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }:
New config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:05.019780 31534 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.17: config changed from index 16 to 17, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:05.023268 6637 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.17: config changed from index 16 to 17, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:05.023880 6768 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 NON_PARTICIPANT]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:05.029371 6769 raft_consensus.cc:1467] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240501 08:20:05.030058 6769 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Committing config change with OpId 3.17: config changed from index 16 to 17, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
W20240501 08:20:05.031219 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:05.033123 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with cas_config_opid_index 16: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:05.035431 6497 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 16 to 17, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
W20240501 08:20:05.038759 6484 catalog_manager.cc:4804] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): delete failed for tablet 30b4f075057e42d0bb3f8cdabdfbc643 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 15 but the committed config has opid_index of 17
I20240501 08:20:05.050439 7028 raft_consensus.cc:1203] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13] Dedup: 3.13->[]
W20240501 08:20:05.057341 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:05.146430 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:05.148954 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d): 4834 bytes on disk
I20240501 08:20:05.149701 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 08:20:05.227669 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:05.228721 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:05.229599 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:05.254863 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:20:05.285593 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:05.288430 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:05.309866 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:05.336293 6896 raft_consensus.cc:1203] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 08:20:05.362835 9011 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskDw_B8y/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 08:14:08.928894 9015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:08.929334 9015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:08.963006 9015 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:14:12.984072 9015 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:14:15.089917 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 314bb13099e14e298e69260ae58eb62c: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:15.096098 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:15.102881 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:15.103083 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:15.103178 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 314bb13099e14e298e69260ae58eb62c: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:15.103340 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:15.103438 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:15.103613 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:15.103708 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:15.103852 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:15.103976 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:15.104132 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:15.104203 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 314bb13099e14e298e69260ae58eb62c: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:15.104285 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:15.104375 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:15.104487 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 56ff53c329aa409a8b7dd792d8aac79b: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:14:15.585250 9015 rebalancer_tool.cc:1084] tablet 070e927b5d684367ae0b40b12a23da54: '97cdee0bf9d84645a1451f9822c5d699' -> 'aba7c226aaca4e8c90931f547d48aaa3' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 08:14:15.585525 9015 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:14:16.664965 10194 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableSchema -> {remote=127.25.107.62:36223, user_credentials={real_user=slave}} blocked reactor thread for 45343.3us
W20240501 08:14:17.743986 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:17.744249 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:17.744589 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:17.744735 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:17.744992 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:17.745173 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:17.745314 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:17.745524 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 46df7d4f86e644a0818048a28715a3d1: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:17.745751 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:17.745873 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:17.745975 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:17.746102 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:17.746207 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 0a814bba1abe4653bcd418a83ec42370
W20240501 08:14:17.746366 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 314bb13099e14e298e69260ae58eb62c: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:17.849134 9015 rebalancer_tool.cc:1074] tablet 5f090fda814f455a88dccd6680bdd05f: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:18.342770 9015 rebalancer_tool.cc:1074] tablet 30b4f075057e42d0bb3f8cdabdfbc643: 'cd3434d87b3c46c6848a711cdf6b150c' -> '0a814bba1abe4653bcd418a83ec42370' move scheduled
I20240501 08:14:18.520565 9015 rebalancer_tool.cc:1084] tablet 193de3c11ae54408a64ae3007bf88920: 'e3a1c4a5b44340be8a183d5d649b27ac' -> 'a518ca5e9de24204bd028b0ce4a77d58' move ignored: Illegal state: tablet 193de3c11ae54408a64ae3007bf88920: replica a518ca5e9de24204bd028b0ce4a77d58 is already present
I20240501 08:14:18.520716 9015 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:14:20.438979 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:20.439306 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e0b03dcc297349f0a696934f63223c7d: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:20.439399 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d20d73b36fc45f6b6695b2a42acb923: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:14:20.439502 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 38f32905d7f144068b86f5b2602e5703: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:20.439590 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 016dbaa82e514f4cb4d67fdb26514c44: could not find any suitable replica to move from server 28b7613bd4834f3ab245c82dd7e80240 to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:20.439710 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f76cd8106b184013b0bd86efbd9321a6: could not find any suitable replica to move from server 97cdee0bf9d84645a1451f9822c5d699 to server 19fdb76c8d1f41d7842e397adfc1ef3f
W20240501 08:14:20.439795 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a24f4747208b4f7e9f25ac196652ebd4: could not find any suitable replica to move from server cd3434d87b3c46c6848a711cdf6b150c to server a518ca5e9de24204bd028b0ce4a77d58
W20240501 08:14:20.440034 9015 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 934448e2027044f3ae7dc3eec4d3f3dd: could not find any suitable replica to move from server e3a1c4a5b44340be8a183d5d649b27ac to server a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:14:20.742678 9015 rebalancer_tool.cc:1074] tablet 491674dfbf74498e8a787bebaa5e95da: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:21.171710 9015 rebalancer_tool.cc:1074] tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '0a814bba1abe4653bcd418a83ec42370' move scheduled
I20240501 08:14:21.760617 9015 rebalancer_tool.cc:1074] tablet 45d076303ea24d7fa0fed5a9d132019c: 'cd3434d87b3c46c6848a711cdf6b150c' -> 'aba7c226aaca4e8c90931f547d48aaa3' move scheduled
I20240501 08:14:21.935464 9015 rebalancer_tool.cc:1074] tablet e9629fbbb4554edc9ba341d42b51f252: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:22.188889 9015 rebalancer_tool.cc:1074] tablet fdaad8e1d86942ddbc97958dcf31b344: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:22.323379 9015 rebalancer_tool.cc:1074] tablet 29506d751571401f86bced7e6f5415d7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:22.790695 9015 rebalancer_tool.cc:1074] tablet dc19d80bad46460087c143778a4b0638: 'cd3434d87b3c46c6848a711cdf6b150c' -> 'aba7c226aaca4e8c90931f547d48aaa3' move scheduled
I20240501 08:14:32.160637 9015 rebalancer_tool.cc:1136] tablet 491674dfbf74498e8a787bebaa5e95da: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:14:32.247869 9015 rebalancer_tool.cc:1136] tablet 45d076303ea24d7fa0fed5a9d132019c: 'cd3434d87b3c46c6848a711cdf6b150c' -> 'aba7c226aaca4e8c90931f547d48aaa3' move completed: Incomplete: tablet 45d076303ea24d7fa0fed5a9d132019c, TS cd3434d87b3c46c6848a711cdf6b150c -> TS aba7c226aaca4e8c90931f547d48aaa3 move failed, target replica disappeared
I20240501 08:14:39.167321 9015 rebalancer_tool.cc:1074] tablet 57ecbb30f7764fceb008baaa2c37da8c: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:43.206241 9015 rebalancer_tool.cc:1136] tablet fdaad8e1d86942ddbc97958dcf31b344: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:14:43.238659 9015 rebalancer_tool.cc:1136] tablet 30b4f075057e42d0bb3f8cdabdfbc643: 'cd3434d87b3c46c6848a711cdf6b150c' -> '0a814bba1abe4653bcd418a83ec42370' move completed: OK
I20240501 08:14:43.299402 9015 rebalancer_tool.cc:1136] tablet 29506d751571401f86bced7e6f5415d7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:43.454169 9015 rebalancer_tool.cc:1136] tablet 5f090fda814f455a88dccd6680bdd05f: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:14:43.540441 9015 rebalancer_tool.cc:1136] tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '0a814bba1abe4653bcd418a83ec42370' move completed: OK
I20240501 08:14:43.699920 9015 rebalancer_tool.cc:1136] tablet dc19d80bad46460087c143778a4b0638: 'cd3434d87b3c46c6848a711cdf6b150c' -> 'aba7c226aaca4e8c90931f547d48aaa3' move completed: OK
I20240501 08:14:43.764659 9015 rebalancer_tool.cc:1136] tablet e9629fbbb4554edc9ba341d42b51f252: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:14:44.194119 9015 rebalancer_tool.cc:1074] tablet 4a6efa669c1645aaa971d59e5d38c8d5: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:44.382453 9015 rebalancer_tool.cc:1074] tablet a9e70954aa4e4fcfbad12e8bfe3e0059: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:44.458643 9015 rebalancer_tool.cc:1074] tablet cfc0c7b429754d92b4dd166c073ac8ed: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:44.602381 9015 rebalancer_tool.cc:1074] tablet 6575b94bd51546f1bd3bb862f012e7ca: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:44.684309 9015 rebalancer_tool.cc:1136] tablet 6575b94bd51546f1bd3bb862f012e7ca: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:14:46.719655 9015 rebalancer_tool.cc:1136] tablet a9e70954aa4e4fcfbad12e8bfe3e0059: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
I20240501 08:14:46.942224 9015 rebalancer_tool.cc:1136] tablet 4a6efa669c1645aaa971d59e5d38c8d5: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:14:47.039721 9015 rebalancer_tool.cc:1136] tablet 57ecbb30f7764fceb008baaa2c37da8c: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:14:47.453894 9015 rebalancer_tool.cc:1074] tablet 399d1a392b3f4d9e8531b6e0b5ad2a61: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:47.545022 9015 rebalancer_tool.cc:1074] tablet f93c2d0f9c7947bb83a50c7fb8c3518a: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:14:47.689805 9015 rebalancer_tool.cc:1074] tablet 8b7f87b6032d45c5be5d748ab2cc71e2: '28b7613bd4834f3ab245c82dd7e80240' -> '' move scheduled
I20240501 08:14:47.753065 9015 rebalancer_tool.cc:1074] tablet 049090443eaa4b18ba90cb60af9a87ab: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:14:55.334022 9015 rebalancer_tool.cc:1123] tablet f93c2d0f9c7947bb83a50c7fb8c3518a: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move is abandoned: Not found: Server with UUID e3a1c4a5b44340be8a183d5d649b27ac not a member of the config. RaftConfig: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:14:55.365381 9015 rebalancer_tool.cc:1136] tablet 399d1a392b3f4d9e8531b6e0b5ad2a61: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
I20240501 08:14:55.465293 9015 rebalancer_tool.cc:1136] tablet cfc0c7b429754d92b4dd166c073ac8ed: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:14:55.608501 9015 rebalancer_tool.cc:1074] tablet d7903106d7e74e1ea102b1abdbfce63e: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move scheduled
I20240501 08:14:55.654804 9015 rebalancer_tool.cc:1074] tablet 070e927b5d684367ae0b40b12a23da54: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:15:02.308727 9015 rebalancer_tool.cc:1136] tablet 8b7f87b6032d45c5be5d748ab2cc71e2: '28b7613bd4834f3ab245c82dd7e80240' -> '' move completed: OK
I20240501 08:15:02.327526 9015 rebalancer_tool.cc:1074] tablet 60238162349647c681685349c6f8ede4: '97cdee0bf9d84645a1451f9822c5d699' -> '' move scheduled
I20240501 08:15:03.127733 9015 rebalancer_tool.cc:1123] tablet 070e927b5d684367ae0b40b12a23da54: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:15:04.122706 9015 rebalancer_tool.cc:1136] tablet 049090443eaa4b18ba90cb60af9a87ab: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
I20240501 08:15:04.140067 9015 rebalancer_tool.cc:1074] tablet 193de3c11ae54408a64ae3007bf88920: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move scheduled
I20240501 08:15:04.152699 9015 rebalancer_tool.cc:1136] tablet 193de3c11ae54408a64ae3007bf88920: 'e3a1c4a5b44340be8a183d5d649b27ac' -> '' move completed: OK
I20240501 08:15:04.177924 9015 rebalancer_tool.cc:1136] tablet 60238162349647c681685349c6f8ede4: '97cdee0bf9d84645a1451f9822c5d699' -> '' move completed: OK
W20240501 08:17:44.151257 10406 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.25.107.62:36223, user_credentials={real_user=slave}} blocked reactor thread for 46303.7us
I20240501 08:20:04.257732 9015 rebalancer_tool.cc:1136] tablet d7903106d7e74e1ea102b1abdbfce63e: 'cd3434d87b3c46c6848a711cdf6b150c' -> '' move completed: OK
Illegal state: tablet server 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): unacceptable health status UNAVAILABLE
I20240501 08:20:05.500674 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:05.513809 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:05.519702 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:05.543628 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:05.560019 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:05.596891 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:05.799681 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:05.962147 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:05.971937 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:05.978613 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:05.981343 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:05.993523 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:06.013201 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:06.033905 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:06.071141 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 193de3c11ae54408a64ae3007bf88920: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:06.096496 6768 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:06.098445 8957 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 08:20:06.099396 30980 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:20:06.102344 31686 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:06.103034 8957 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:06.108870 6496 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 9 to 10, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:06.134927 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet a7a56c284444496b8579600a49f19d1d with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:37122
I20240501 08:20:06.136096 6768 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:06.138528 31695 tablet_replica.cc:330] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:20:06.138872 31695 raft_consensus.cc:2229] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:20:06.138880 8957 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 08:20:06.139585 31686 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:20:06.139711 31695 raft_consensus.cc:2258] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:20:06.140542 31695 ts_tablet_manager.cc:1875] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:20:06.140594 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): Couldn't send request to peer e3a1c4a5b44340be8a183d5d649b27ac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:06.142392 30980 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:06.143818 8957 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:06.147321 6484 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a7a56c284444496b8579600a49f19d1d with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:06.153388 6495 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 10 to 11, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } }
I20240501 08:20:06.158233 31695 ts_tablet_manager.cc:1888] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 08:20:06.158422 31695 log.cc:1196] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7a56c284444496b8579600a49f19d1d
I20240501 08:20:06.159160 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet a7a56c284444496b8579600a49f19d1d (table rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]) successfully deleted
I20240501 08:20:06.279431 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:06.286866 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:06.302414 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:06.311995 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:06.318292 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:20:06.355609 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 08:20:06.366559 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:06.385208 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:20:06.403228 6896 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.17, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:06.405084 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 17. Preceding OpId from leader: term: 3 index: 18. (index mismatch)
I20240501 08:20:06.405658 31389 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240501 08:20:06.408272 31752 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.18: config changed from index 17 to 18, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:06.408810 6637 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.18: config changed from index 17 to 18, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:06.413726 6493 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 17 to 18, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:06.419265 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:40900
I20240501 08:20:06.419925 6896 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:06.420544 31758 tablet_replica.cc:330] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:20:06.420779 31758 raft_consensus.cc:2229] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:20:06.421065 31758 raft_consensus.cc:2258] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:20:06.422310 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20240501 08:20:06.422698 31758 ts_tablet_manager.cc:1875] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:20:06.422783 6835 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): Couldn't send request to peer cd3434d87b3c46c6848a711cdf6b150c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:06.423112 31389 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20240501 08:20:06.425530 31389 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.19: config changed from index 18 to 19, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:06.425944 6637 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.19: config changed from index 18 to 19, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:06.430387 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with cas_config_opid_index 18: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:06.432018 6493 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 18 to 19, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:06.438781 31758 ts_tablet_manager.cc:1888] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.17
I20240501 08:20:06.438918 31758 log.cc:1196] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:20:06.440016 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 30b4f075057e42d0bb3f8cdabdfbc643 (table rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]) successfully deleted
W20240501 08:20:06.541538 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:06.560750 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:06.562175 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:06.578558 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:06.592593 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:06.594380 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:06.746098 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:06.811535 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:06.815387 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:06.815820 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:06.822342 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240501 08:20:06.835633 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:06.853296 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240501 08:20:06.867998 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:06.914072 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:06.964210 8893 consensus_peers.cc:485] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:06.982455 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:20:07.038550 31863 ts_tablet_manager.cc:902] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
W20240501 08:20:07.069079 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:07.075188 31863 tablet_copy_client.cc:308] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:20:07.075942 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet a7a56c284444496b8579600a49f19d1d from peer e3a1c4a5b44340be8a183d5d649b27ac ({username='slave'} at 127.25.107.3:42535)
I20240501 08:20:07.076102 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet a7a56c284444496b8579600a49f19d1d from peer e3a1c4a5b44340be8a183d5d649b27ac at {username='slave'} at 127.25.107.3:42535: session id = e3a1c4a5b44340be8a183d5d649b27ac-a7a56c284444496b8579600a49f19d1d
W20240501 08:20:07.077183 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:07.078583 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240501 08:20:07.079648 6789 tablet_copy_source_session.cc:205] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:07.080469 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:07.084815 31863 ts_tablet_manager.cc:1875] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:20:07.089079 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:07.102564 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:07.109969 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:07.120844 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:07.122303 31863 ts_tablet_manager.cc:1888] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240501 08:20:07.122593 31863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a56c284444496b8579600a49f19d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:07.134352 31863 tablet_copy_client.cc:783] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:07.143823 31863 tablet_copy_client.cc:647] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:07.145517 31863 tablet_copy_client.cc:523] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 08:20:07.146901 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:07.147035 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:20:07.154230 31863 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap starting.
W20240501 08:20:07.162797 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:07.164273 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:07.187570 31863 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:07.188045 31863 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Bootstrap complete.
I20240501 08:20:07.188357 31863 ts_tablet_manager.cc:1367] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.009s
I20240501 08:20:07.189167 31863 raft_consensus.cc:348] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:07.189406 31863 raft_consensus.cc:729] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a1c4a5b44340be8a183d5d649b27ac, State: Initialized, Role: FOLLOWER
I20240501 08:20:07.189749 31863 consensus_queue.cc:260] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:07.190480 31863 ts_tablet_manager.cc:1398] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:20:07.191996 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-a7a56c284444496b8579600a49f19d1d received from {username='slave'} at 127.25.107.3:42535
I20240501 08:20:07.192127 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session e3a1c4a5b44340be8a183d5d649b27ac-a7a56c284444496b8579600a49f19d1d on tablet a7a56c284444496b8579600a49f19d1d with peer e3a1c4a5b44340be8a183d5d649b27ac
W20240501 08:20:07.195503 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:20:07.272081 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:07.286932 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:20:07.336246 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:07.338682 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:07.345073 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:07.351056 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:07.368458 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:07.371802 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:07.408739 6945 maintenance_manager.cc:419] P e3a1c4a5b44340be8a183d5d649b27ac: Scheduling UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d): 4834 bytes on disk
I20240501 08:20:07.409466 6836 maintenance_manager.cc:643] P e3a1c4a5b44340be8a183d5d649b27ac: UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240501 08:20:07.423019 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:20:07.453784 31926 ts_tablet_manager.cc:902] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
W20240501 08:20:07.455236 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:20:07.459021 31926 tablet_copy_client.cc:308] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.3:43505
I20240501 08:20:07.459635 6919 tablet_copy_service.cc:140] P e3a1c4a5b44340be8a183d5d649b27ac: Received BeginTabletCopySession request for tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer cd3434d87b3c46c6848a711cdf6b150c ({username='slave'} at 127.25.107.2:34709)
I20240501 08:20:07.459789 6919 tablet_copy_service.cc:161] P e3a1c4a5b44340be8a183d5d649b27ac: Beginning new tablet copy session on tablet 30b4f075057e42d0bb3f8cdabdfbc643 from peer cd3434d87b3c46c6848a711cdf6b150c at {username='slave'} at 127.25.107.2:34709: session id = cd3434d87b3c46c6848a711cdf6b150c-30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:20:07.463302 6919 tablet_copy_source_session.cc:205] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:07.464393 31926 ts_tablet_manager.cc:1875] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:20:07.469599 31926 ts_tablet_manager.cc:1888] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.17
I20240501 08:20:07.469866 31926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30b4f075057e42d0bb3f8cdabdfbc643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:07.472909 31926 tablet_copy_client.cc:783] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:07.479297 31926 tablet_copy_client.cc:647] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:07.481184 31926 tablet_copy_client.cc:523] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:07.484450 31926 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap starting.
I20240501 08:20:07.513121 31926 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:07.513617 31926 tablet_bootstrap.cc:492] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Bootstrap complete.
I20240501 08:20:07.513934 31926 ts_tablet_manager.cc:1367] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Time spent bootstrapping tablet: real 0.030s user 0.029s sys 0.000s
I20240501 08:20:07.514791 31926 raft_consensus.cc:348] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:07.515013 31926 raft_consensus.cc:729] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3434d87b3c46c6848a711cdf6b150c, State: Initialized, Role: FOLLOWER
I20240501 08:20:07.515275 31926 consensus_queue.cc:260] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 3.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:07.515926 31926 ts_tablet_manager.cc:1398] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:20:07.516837 6919 tablet_copy_service.cc:342] P e3a1c4a5b44340be8a183d5d649b27ac: Request end of tablet copy session cd3434d87b3c46c6848a711cdf6b150c-30b4f075057e42d0bb3f8cdabdfbc643 received from {username='slave'} at 127.25.107.2:34709
I20240501 08:20:07.516975 6919 tablet_copy_service.cc:433] P e3a1c4a5b44340be8a183d5d649b27ac: ending tablet copy session cd3434d87b3c46c6848a711cdf6b150c-30b4f075057e42d0bb3f8cdabdfbc643 on tablet 30b4f075057e42d0bb3f8cdabdfbc643 with peer cd3434d87b3c46c6848a711cdf6b150c
I20240501 08:20:07.542806 6898 raft_consensus.cc:1203] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240501 08:20:07.547974 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:07.587533 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:07.590390 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:07.597268 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:07.598927 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:07.600026 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:07.683324 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:20:07.684191 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:07.693976 6815 maintenance_manager.cc:419] P cd3434d87b3c46c6848a711cdf6b150c: Scheduling UndoDeltaBlockGCOp(30b4f075057e42d0bb3f8cdabdfbc643): 4922 bytes on disk
I20240501 08:20:07.694705 6706 maintenance_manager.cc:643] P cd3434d87b3c46c6848a711cdf6b150c: UndoDeltaBlockGCOp(30b4f075057e42d0bb3f8cdabdfbc643) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240501 08:20:07.768553 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:20:07.799510 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:07.813899 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:07.814843 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:07.815456 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:07.822645 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240501 08:20:07.834805 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:08.011237 6768 raft_consensus.cc:1203] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.18->[3.19-3.19] Dedup: 3.19->[]
W20240501 08:20:08.067970 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:08.229954 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:08.231323 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:08.232625 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:08.242463 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:08.245487 31998 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:20:08.245689 31998 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:20:08.246392 31998 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:20:08.256033 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:20:08.257330 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:08.258975 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:08.259088 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:20:08.259351 31998 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:08.264572 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:08.330243 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:20:08.349146 6768 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:08.350950 8957 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 08:20:08.351472 31998 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:20:08.353940 31828 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:08.354648 8957 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:08.360374 6490 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 11 to 12, VOTER e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3) evicted. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:08.366894 6877 tablet_service.cc:1514] Processing DeleteTablet for tablet a7a56c284444496b8579600a49f19d1d with delete_type TABLET_DATA_TOMBSTONED (TS e3a1c4a5b44340be8a183d5d649b27ac not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:37122
I20240501 08:20:08.367714 6768 consensus_queue.cc:237] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:08.368907 32066 tablet_replica.cc:330] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: stopping tablet replica
I20240501 08:20:08.369136 32066 raft_consensus.cc:2229] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:20:08.369326 32066 raft_consensus.cc:2258] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:20:08.369997 8957 raft_consensus.cc:1261] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Refusing update from remote peer cd3434d87b3c46c6848a711cdf6b150c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 08:20:08.370695 31998 consensus_queue.cc:1035] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 08:20:08.370961 32066 ts_tablet_manager.cc:1875] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:20:08.371349 6702 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193): Couldn't send request to peer 97cdee0bf9d84645a1451f9822c5d699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a56c284444496b8579600a49f19d1d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:08.373355 31998 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:08.373989 8957 raft_consensus.cc:2932] T a7a56c284444496b8579600a49f19d1d P 19fdb76c8d1f41d7842e397adfc1ef3f [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:08.379777 6484 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a7a56c284444496b8579600a49f19d1d with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240501 08:20:08.380234 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 08:20:08.380776 6490 catalog_manager.cc:5484] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c reported cstate change: config changed from index 12 to 13, VOTER 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1) added. New cstate: current_term: 1 leader_uuid: "cd3434d87b3c46c6848a711cdf6b150c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:08.383257 32066 ts_tablet_manager.cc:1888] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 08:20:08.383414 32066 log.cc:1196] T a7a56c284444496b8579600a49f19d1d P e3a1c4a5b44340be8a183d5d649b27ac: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7a56c284444496b8579600a49f19d1d
I20240501 08:20:08.384392 6481 catalog_manager.cc:4830] TS e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505): tablet a7a56c284444496b8579600a49f19d1d (table rebalance_test_table_4 [id=4d20d73b36fc45f6b6695b2a42acb923]) successfully deleted
I20240501 08:20:08.508787 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:08.530970 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:08.533903 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:08.540621 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:08.553560 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:08.615450 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240501 08:20:08.655535 6898 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 3.19, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:08.657472 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 3 index: 20. (index mismatch)
I20240501 08:20:08.658031 31389 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20240501 08:20:08.660948 32122 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.20: config changed from index 19 to 20, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:08.661645 6637 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.20: config changed from index 19 to 20, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
W20240501 08:20:08.670279 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:08.673002 6495 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 reported cstate change: config changed from index 19 to 20, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:08.678519 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 20) from {username='slave'} at 127.0.0.1:40900
I20240501 08:20:08.684316 6898 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 3.20, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:08.686337 6768 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 3 index: 21. (index mismatch)
I20240501 08:20:08.686532 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 3 index: 21. (index mismatch)
I20240501 08:20:08.687122 32122 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20240501 08:20:08.687422 31389 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20240501 08:20:08.691423 6768 raft_consensus.cc:858] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }.
Pending config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }:
New config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
W20240501 08:20:08.691568 6768 raft_consensus.cc:1565] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 NON_PARTICIPANT]: Could not prepare op 'term: 3 index: 21' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }.
Pending config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }:
New config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } }
I20240501 08:20:08.704916 32122 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.21: config changed from index 20 to 21, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
W20240501 08:20:08.711872 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d7903106d7e74e1ea102b1abdbfce63e: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:20:08.718669 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with cas_config_opid_index 20: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:08.719802 6637 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.21: config changed from index 20 to 21, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:08.720916 6495 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 20 to 21, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
I20240501 08:20:08.722827 6768 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 NON_PARTICIPANT]: Committing config change with OpId 3.20: config changed from index 19 to 20, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
W20240501 08:20:08.729565 6484 catalog_manager.cc:4804] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): delete failed for tablet 30b4f075057e42d0bb3f8cdabdfbc643 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 19 but the committed config has opid_index of 20
I20240501 08:20:08.732213 6768 raft_consensus.cc:1467] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240501 08:20:08.733110 6768 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Committing config change with OpId 3.21: config changed from index 20 to 21, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } }
W20240501 08:20:08.753273 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:08.757530 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:08.866097 32163 ts_tablet_manager.cc:902] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699: Initiating tablet copy from peer cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507)
I20240501 08:20:08.868489 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:08.880630 32163 tablet_copy_client.cc:308] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Beginning tablet copy session from remote peer at address 127.25.107.2:46507
I20240501 08:20:08.881484 6789 tablet_copy_service.cc:140] P cd3434d87b3c46c6848a711cdf6b150c: Received BeginTabletCopySession request for tablet a7a56c284444496b8579600a49f19d1d from peer 97cdee0bf9d84645a1451f9822c5d699 ({username='slave'} at 127.25.107.1:43289)
I20240501 08:20:08.881655 6789 tablet_copy_service.cc:161] P cd3434d87b3c46c6848a711cdf6b150c: Beginning new tablet copy session on tablet a7a56c284444496b8579600a49f19d1d from peer 97cdee0bf9d84645a1451f9822c5d699 at {username='slave'} at 127.25.107.1:43289: session id = 97cdee0bf9d84645a1451f9822c5d699-a7a56c284444496b8579600a49f19d1d
I20240501 08:20:08.882537 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:08.888489 6789 tablet_copy_source_session.cc:205] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:20:08.890152 32163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a56c284444496b8579600a49f19d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:20:08.898483 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:08.901084 32163 tablet_copy_client.cc:783] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Starting download of 5 data blocks...
I20240501 08:20:08.909621 32163 tablet_copy_client.cc:647] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Starting download of 1 WAL segments...
I20240501 08:20:08.911489 32163 tablet_copy_client.cc:523] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:20:08.914932 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:08.916983 32163 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap starting.
I20240501 08:20:08.921792 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:08.968310 32163 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:20:08.968874 32163 tablet_bootstrap.cc:492] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699: Bootstrap complete.
I20240501 08:20:08.969252 32163 ts_tablet_manager.cc:1367] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699: Time spent bootstrapping tablet: real 0.052s user 0.024s sys 0.003s
I20240501 08:20:08.970048 32163 raft_consensus.cc:348] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:08.970309 32163 raft_consensus.cc:729] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cdee0bf9d84645a1451f9822c5d699, State: Initialized, Role: FOLLOWER
I20240501 08:20:08.970561 32163 consensus_queue.cc:260] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:08.971072 32163 ts_tablet_manager.cc:1398] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:20:08.971803 6789 tablet_copy_service.cc:342] P cd3434d87b3c46c6848a711cdf6b150c: Request end of tablet copy session 97cdee0bf9d84645a1451f9822c5d699-a7a56c284444496b8579600a49f19d1d received from {username='slave'} at 127.25.107.1:43289
I20240501 08:20:08.971912 6789 tablet_copy_service.cc:433] P cd3434d87b3c46c6848a711cdf6b150c: ending tablet copy session 97cdee0bf9d84645a1451f9822c5d699-a7a56c284444496b8579600a49f19d1d on tablet a7a56c284444496b8579600a49f19d1d with peer 97cdee0bf9d84645a1451f9822c5d699
I20240501 08:20:08.995730 6684 maintenance_manager.cc:419] P 97cdee0bf9d84645a1451f9822c5d699: Scheduling UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d): 4834 bytes on disk
I20240501 08:20:08.996376 6575 maintenance_manager.cc:643] P 97cdee0bf9d84645a1451f9822c5d699: UndoDeltaBlockGCOp(a7a56c284444496b8579600a49f19d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240501 08:20:09.073351 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:09.106525 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:09.124377 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:09.138916 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:09.146723 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:09.149281 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:09.197511 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:09.200863 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:09.243954 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240501 08:20:09.281493 6637 raft_consensus.cc:1203] T a7a56c284444496b8579600a49f19d1d P 97cdee0bf9d84645a1451f9822c5d699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
W20240501 08:20:09.394954 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:09.406853 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 08:20:09.408026 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:09.420207 8893 consensus_peers.cc:485] T 049090443eaa4b18ba90cb60af9a87ab P 19fdb76c8d1f41d7842e397adfc1ef3f -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:09.457388 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:09.486009 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:09.545451 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 08:20:09.547588 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:09.592235 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:09.592563 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:09.602797 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:09.607501 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:09.608263 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:09.610291 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:09.618368 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:09.618786 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:09.725886 6898 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 3.21, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } }
I20240501 08:20:09.727705 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 21. Preceding OpId from leader: term: 3 index: 22. (index mismatch)
I20240501 08:20:09.728125 31389 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20240501 08:20:09.730551 32274 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.22: config changed from index 21 to 22, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:09.731079 6637 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.22: config changed from index 21 to 22, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:09.735736 6493 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 21 to 22, VOTER cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2) evicted. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } }
I20240501 08:20:09.742175 6748 tablet_service.cc:1514] Processing DeleteTablet for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with delete_type TABLET_DATA_TOMBSTONED (TS cd3434d87b3c46c6848a711cdf6b150c not found in new config with opid_index 22) from {username='slave'} at 127.0.0.1:40900
I20240501 08:20:09.742786 32280 tablet_replica.cc:330] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: stopping tablet replica
I20240501 08:20:09.743057 32280 raft_consensus.cc:2229] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:20:09.743448 6898 consensus_queue.cc:237] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 3.22, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } }
I20240501 08:20:09.744071 32280 raft_consensus.cc:2258] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:20:09.745496 6637 raft_consensus.cc:1261] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Refusing update from remote peer e3a1c4a5b44340be8a183d5d649b27ac: Log matching property violated. Preceding OpId in replica: term: 3 index: 22. Preceding OpId from leader: term: 3 index: 23. (index mismatch)
I20240501 08:20:09.746158 31389 consensus_queue.cc:1035] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240501 08:20:09.746217 32280 ts_tablet_manager.cc:1875] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:20:09.746901 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:09.747665 6833 consensus_peers.cc:485] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30b4f075057e42d0bb3f8cdabdfbc643. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:09.749181 31389 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac [term 3 LEADER]: Committing config change with OpId 3.23: config changed from index 22 to 23, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:09.752641 6637 raft_consensus.cc:2932] T 30b4f075057e42d0bb3f8cdabdfbc643 P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Committing config change with OpId 3.23: config changed from index 22 to 23, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
I20240501 08:20:09.754813 32280 ts_tablet_manager.cc:1888] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.21
I20240501 08:20:09.754985 32280 log.cc:1196] T 30b4f075057e42d0bb3f8cdabdfbc643 P cd3434d87b3c46c6848a711cdf6b150c: Deleting WAL directory at /tmp/dist-test-taskDw_B8y/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714551039513895-26028-0/raft_consensus-itest-cluster/ts-1/wal/wals/30b4f075057e42d0bb3f8cdabdfbc643
I20240501 08:20:09.755931 6484 catalog_manager.cc:4830] TS cd3434d87b3c46c6848a711cdf6b150c (127.25.107.2:46507): tablet 30b4f075057e42d0bb3f8cdabdfbc643 (table rebalance_test_table_2 [id=f76cd8106b184013b0bd86efbd9321a6]) successfully deleted
I20240501 08:20:09.756502 6481 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 30b4f075057e42d0bb3f8cdabdfbc643 with cas_config_opid_index 22: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:20:09.758697 6489 catalog_manager.cc:5484] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac reported cstate change: config changed from index 22 to 23, VOTER 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) added. New cstate: current_term: 3 leader_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } }
W20240501 08:20:09.782366 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:09.792503 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:09.832520 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 08:20:09.876605 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:09.878355 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:09.889678 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:09.899972 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:09.917933 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:09.922493 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:09.969552 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:10.084476 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 08:20:10.107860 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:10.141470 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:20:10.142431 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:10.154742 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:10.157647 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:10.175906 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:10.177712 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:10.215066 6481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20240501 08:20:10.221591 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 193de3c11ae54408a64ae3007bf88920: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:10.369482 32397 ts_tablet_manager.cc:902] T 30b4f075057e42d0bb3f8cdabdfbc643 P 19fdb76c8d1f41d7842e397adfc1ef3f: Initiating tablet copy from peer e3a1c4a5b44340be8a183d5d649b27ac (127.25.107.3:43505)
F20240501 08:20:10.369892 32397 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.queue_time_us, tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.queue_time_us) (tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.run_cpu_time_us, tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.run_cpu_time_us) (tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.run_wall_time_us, tablet-download-pool-049090443eaa4b18ba90cb60af9a87ab.run_wall_time_us) (tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.queue_time_us, tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.queue_time_us) (tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_cpu_time_us, tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_cpu_time_us) (tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_wall_time_us, tablet-download-pool-13b4e2d5fe4846e8b3c6081abb3c53da.run_wall_time_us) (tablet-download-pool-30b4f075057e42d0bb3f8cdabdfbc643.queue_time_us, tablet-download-pool-30b4f075057e42d0bb3f8cdabdfbc643.queue_time_us) (tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.queue_time_us, tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.queue_time_us) (tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.run_cpu_time_us, tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.run_cpu_time_us) (tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.run_wall_time_us, tablet-download-pool-399d1a392b3f4d9e8531b6e0b5ad2a61.run_wall_time_us) (tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.queue_time_us, tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.queue_time_us) (tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.run_cpu_time_us, tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.run_cpu_time_us) (tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.run_wall_time_us, tablet-download-pool-491674dfbf74498e8a787bebaa5e95da.run_wall_time_us) (tablet-download-pool-4a6efa669c1645aaa971d59e5d38c8d5.queue_time_us, tablet-download-pool-4a6efa669c1645aaa971d59e5d38c8d5.queue_time_us) (tablet-download-pool-4a6efa669c1645aaa971d59e5d38c8d5.run_cpu_time_us, tablet-download-pool-4a6efa669c1645aaa971d59e5d38c8d5.run_cpu_time_us) (tablet-download-pool-4a6efa669c1645aaa971d59e5d38c8d5.run_wall_time_us, tablet-download-pool-4a6efa669c1645aaa971d59e5d38c8d5.run_wall_time_us) (tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.queue_time_us, tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.queue_time_us) (tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_cpu_time_us, tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_cpu_time_us) (tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_wall_time_us, tablet-download-pool-57ecbb30f7764fceb008baaa2c37da8c.run_wall_time_us) (tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.queue_time_us, tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.queue_time_us) (tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.run_cpu_time_us, tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.run_cpu_time_us) (tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.run_wall_time_us, tablet-download-pool-5f090fda814f455a88dccd6680bdd05f.run_wall_time_us) (tablet-download-pool-60238162349647c681685349c6f8ede4.queue_time_us, tablet-download-pool-60238162349647c681685349c6f8ede4.queue_time_us) (tablet-download-pool-60238162349647c681685349c6f8ede4.run_cpu_time_us, tablet-download-pool-60238162349647c681685349c6f8ede4.run_cpu_time_us) (tablet-download-pool-60238162349647c681685349c6f8ede4.run_wall_time_us, tablet-download-pool-60238162349647c681685349c6f8ede4.run_wall_time_us) (tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.queue_time_us, tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.queue_time_us) (tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.run_cpu_time_us, tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.run_cpu_time_us) (tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.run_wall_time_us, tablet-download-pool-658f2a172a2d47c8bf8b44c9d51132e4.run_wall_time_us) (tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.queue_time_us, tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.queue_time_us) (tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.run_cpu_time_us, tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.run_cpu_time_us) (tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.run_wall_time_us, tablet-download-pool-8b7f87b6032d45c5be5d748ab2cc71e2.run_wall_time_us) (tablet-download-pool-a7a56c284444496b8579600a49f19d1d.queue_time_us, tablet-download-pool-a7a56c284444496b8579600a49f19d1d.queue_time_us) (tablet-download-pool-a7a56c284444496b8579600a49f19d1d.run_cpu_time_us, tablet-download-pool-a7a56c284444496b8579600a49f19d1d.run_cpu_time_us) (tablet-download-pool-a7a56c284444496b8579600a49f19d1d.run_wall_time_us, tablet-download-pool-a7a56c284444496b8579600a49f19d1d.run_wall_time_us) (tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.queue_time_us, tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.queue_time_us) (tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_cpu_time_us, tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_cpu_time_us) (tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_wall_time_us, tablet-download-pool-a9e70954aa4e4fcfbad12e8bfe3e0059.run_wall_time_us) (tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.queue_time_us, tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.queue_time_us) (tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.run_cpu_time_us, tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.run_cpu_time_us) (tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.run_wall_time_us, tablet-download-pool-d7903106d7e74e1ea102b1abdbfce63e.run_wall_time_us) (tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.queue_time_us, tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.queue_time_us) (tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.run_cpu_time_us, tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.run_cpu_time_us) (tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.run_wall_time_us, tablet-download-pool-e9629fbbb4554edc9ba341d42b51f252.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fc857b2bdcd google::LogMessage::Fail() at ??:0
@ 0x7fc857b2fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc857b2b7cc google::LogMessage::Flush() at ??:0
@ 0x7fc857b2cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
W20240501 08:20:10.380165 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
@ 0x7fc85a86c742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fc85a838f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240501 08:20:10.388726 8938 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7fc85a832cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fc869c858de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fc869c9ddaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 08:20:10.403973 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7fc869dc8480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fc869e00a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fc869de3981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fc869de370e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fc85a83cb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc85a84019c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
W20240501 08:20:10.434146 8893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 10 similar messages]
@ 0x7fc85a840019 std::_Function_handler<>::_M_invoke() at ??:0
I20240501 08:20:10.436343 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fc85a81b01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc85dba06db start_thread at ??:0
@ 0x7fc8569de71f clone at ??:0
I20240501 08:20:10.463449 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:10.521097 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:10.697765 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:10.759969 6703 connection.cc:535] client connection to 127.25.107.8:41625 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:20:10.759970 6833 connection.cc:535] client connection to 127.25.107.8:41625 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:20:10.759966 6963 connection.cc:535] client connection to 127.25.107.8:41625 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:20:10.760649 6572 connection.cc:535] client connection to 127.25.107.8:41625 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:20:10.761624 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:10.761770 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:10.761857 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:10.761946 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:10.761982 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:10.762137 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:10.762284 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:10.762365 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:10.763366 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:10.793839 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:10.820160 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:10.839737 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:20:10.886950 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:10.889060 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:10.897022 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:10.902457 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:10.959766 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:10.999552 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:11.084750 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:11.091590 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:11.099928 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:11.108134 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:11.154305 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:11.164420 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 08:20:11.217859 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:20:11.273346 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:11.279827 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:11.281456 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:11.284492 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:11.338186 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:11.444962 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:11.496327 32460 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:11.496573 32460 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:11.497354 32460 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:20:11.498890 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:11.499078 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:11.499169 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:20:11.499439 32460 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:11.580966 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:20:11.674441 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:20:11.688803 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:11.708370 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:11.714897 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:11.751008 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:11.768891 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:11.769618 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:11.771677 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:11.850143 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:11.852150 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:11.852483 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 08:20:11.859128 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:11.875118 32424 raft_consensus.cc:482] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:11.875341 32424 raft_consensus.cc:504] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:20:11.876044 32424 leader_election.cc:288] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4:43623)
I20240501 08:20:11.876729 7027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "97cdee0bf9d84645a1451f9822c5d699" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: false dest_uuid: "28b7613bd4834f3ab245c82dd7e80240" is_pre_election: true
W20240501 08:20:11.882966 6572 leader_election.cc:334] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:11.883136 6572 leader_election.cc:302] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97cdee0bf9d84645a1451f9822c5d699; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, 28b7613bd4834f3ab245c82dd7e80240
I20240501 08:20:11.883386 32424 raft_consensus.cc:2726] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:20:11.900246 7008 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:20:11.913864 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240501 08:20:11.918534 32132 raft_consensus.cc:482] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:11.918758 32132 raft_consensus.cc:504] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:20:11.919492 32132 leader_election.cc:288] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
I20240501 08:20:11.920459 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699" is_pre_election: true
I20240501 08:20:11.920675 6637 raft_consensus.cc:2454] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 3.
W20240501 08:20:11.921604 6963 leader_election.cc:334] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:11.922931 6962 leader_election.cc:302] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:20:11.923663 32132 raft_consensus.cc:2781] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 08:20:11.923802 32132 raft_consensus.cc:482] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:11.923899 32132 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:20:11.931041 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:11.933674 32132 raft_consensus.cc:504] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:20:11.935375 6637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f090fda814f455a88dccd6680bdd05f" candidate_uuid: "28b7613bd4834f3ab245c82dd7e80240" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: false dest_uuid: "97cdee0bf9d84645a1451f9822c5d699"
I20240501 08:20:11.935590 6637 raft_consensus.cc:3037] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:20:11.935796 32132 leader_election.cc:288] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 4 election: Requested vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), 97cdee0bf9d84645a1451f9822c5d699 (127.25.107.1:45193)
W20240501 08:20:11.935995 6963 leader_election.cc:334] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:11.938102 6637 raft_consensus.cc:2454] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b7613bd4834f3ab245c82dd7e80240 in term 4.
I20240501 08:20:11.938625 6962 leader_election.cc:302] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28b7613bd4834f3ab245c82dd7e80240, 97cdee0bf9d84645a1451f9822c5d699; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f
I20240501 08:20:11.939949 32132 raft_consensus.cc:2781] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:20:11.940168 32132 raft_consensus.cc:686] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [term 4 LEADER]: Becoming Leader. State: Replica: 28b7613bd4834f3ab245c82dd7e80240, State: Running, Role: LEADER
I20240501 08:20:11.940474 32132 consensus_queue.cc:237] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } }
I20240501 08:20:11.946100 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:11.965045 6493 catalog_manager.cc:5484] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 reported cstate change: term changed from 3 to 4, leader changed from 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8) to 28b7613bd4834f3ab245c82dd7e80240 (127.25.107.4). New cstate: current_term: 4 leader_uuid: "28b7613bd4834f3ab245c82dd7e80240" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 } } peers { permanent_uuid: "28b7613bd4834f3ab245c82dd7e80240" member_type: VOTER last_known_addr { host: "127.25.107.4" port: 43623 } } }
I20240501 08:20:11.965795 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:12.007860 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:12.123311 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:12.179165 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 08:20:12.259083 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:12.271718 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:12.320722 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:12.344625 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:12.356945 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:12.367774 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:12.368461 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:12.372310 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:12.392374 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:12.440834 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 08:20:12.455307 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:12.478852 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:20:12.499647 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:12.530860 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:12.555766 6637 raft_consensus.cc:1261] T 5f090fda814f455a88dccd6680bdd05f P 97cdee0bf9d84645a1451f9822c5d699 [term 4 FOLLOWER]: Refusing update from remote peer 28b7613bd4834f3ab245c82dd7e80240: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20240501 08:20:12.556293 32132 consensus_queue.cc:1035] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cdee0bf9d84645a1451f9822c5d699" member_type: VOTER last_known_addr { host: "127.25.107.1" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 08:20:12.593457 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:12.608986 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:12.609930 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:12.627862 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:12.703904 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 08:20:12.704794 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:12.764931 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:12.804447 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:12.815441 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:12.841270 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:12.914467 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:12.963344 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:12.986151 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:13.025882 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:13.030467 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:13.036285 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:13.045653 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:13.090040 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:13.170306 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:13.242491 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:13.243999 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:13.255035 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:13.255180 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:13.276054 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:13.281391 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:13.314798 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:13.324162 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:13.387140 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:13.395458 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:20:13.428839 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:13.437384 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240501 08:20:13.480803 303 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:13.481007 303 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:13.481696 303 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:20:13.483363 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:13.483572 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:13.483668 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:20:13.483943 303 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:13.512252 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:13.536790 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 08:20:13.550968 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:13.572978 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:13.638144 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:13.641703 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:13.656778 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:13.688750 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:13.713133 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:20:13.853771 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:13.860828 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:13.860981 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:13.862315 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:13.920192 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:13.950146 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:14.108835 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:14.221971 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:20:14.251386 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:14.256366 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:14.263155 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:14.266899 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:14.283444 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:14.285998 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:14.323374 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:14.325893 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:14.359915 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:14.436924 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:14.442046 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:14.443030 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240501 08:20:14.449575 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:14.454461 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:14.535928 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 08:20:14.564898 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:14.615742 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:14.616521 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 08:20:14.625707 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:14.691449 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:14.747920 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:14.839344 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:14.895550 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:20:14.896234 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:14.898586 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:14.903065 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:20:14.922097 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:14.922333 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:14.932893 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:20:14.933938 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:15.084172 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:15.084409 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:15.086406 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:15.100863 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:15.118114 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:15.118975 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:15.137030 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:20:15.174947 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:15.218413 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:15.222508 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:15.272228 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:15.287813 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 08:20:15.375857 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:15.400043 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:15.417320 6484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240501 08:20:15.418556 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:20:15.490142 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:15.511219 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:15.523288 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:15.527053 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:15.544093 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:15.580919 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:15.589219 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:15.647336 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:15.648885 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:15.694900 303 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:15.695086 303 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:15.695721 303 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:20:15.698974 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:15.699206 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:15.699311 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:20:15.699856 303 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:15.709332 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:15.730162 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:15.741171 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:15.742111 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:15.763461 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:15.940959 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:15.956173 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:15.981444 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:16.025602 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:16.036638 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:16.037539 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:16.043923 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:16.044024 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:16.125885 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240501 08:20:16.132340 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:16.142498 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:16.155184 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:16.158015 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:16.247030 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:20:16.336570 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:16.351668 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:16.359558 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:16.364853 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:16.438197 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:16.548722 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:16.614302 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:16.719655 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:20:16.761787 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:16.766130 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:16.785147 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:16.788123 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:16.800330 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:16.837182 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:16.864485 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:16.883082 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 08:20:16.895153 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 08:20:16.913447 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d7903106d7e74e1ea102b1abdbfce63e: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:20:16.951871 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:20:16.985817 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:17.001282 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:17.008049 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:17.008462 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:17.059652 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:17.060662 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:17.061368 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:17.075076 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:17.084951 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:17.205004 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:17.309561 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 08:20:17.382560 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:17.386337 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:17.389946 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:17.406483 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:17.407259 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:17.450165 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:17.461393 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:17.502940 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:20:17.519207 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:20:17.582229 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:17.594763 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:17.594763 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:17.597540 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:17.618501 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:17.624619 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:17.662427 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 08:20:17.724799 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:17.766081 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 08:20:17.766913 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:17.775422 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:17.888840 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:17.894963 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:17.952098 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:17.982628 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:17.997532 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:18.005285 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:18.006502 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:18.040652 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:18.116518 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:18.120131 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:18.137864 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:18.175347 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:18.195390 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:18.198936 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:18.214519 684 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:20:18.214756 684 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:20:18.215471 684 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:20:18.217511 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:20:18.219777 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:18.220395 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:18.220530 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:20:18.221051 684 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:18.225251 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:18.340435 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:18.355531 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:18.425446 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:20:18.459497 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 193de3c11ae54408a64ae3007bf88920: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:18.477891 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:18.509753 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 08:20:18.561185 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:18.592833 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:18.596326 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:18.596575 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:18.600965 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:18.652957 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 08:20:18.742713 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:18.785815 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:18.788005 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:18.805580 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:18.806818 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:18.813711 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:18.831895 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:18.851438 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:18.894654 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:18.950699 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:19.106760 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:19.159826 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:19.163664 910 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:19.163847 910 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:19.164515 910 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:20:19.165908 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:19.166080 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:19.166169 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:20:19.166447 910 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:19.195053 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:19.202322 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:19.234298 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:19.234880 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:19.238512 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:19.239545 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:19.298609 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:19.330637 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240501 08:20:19.413722 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:19.426370 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:19.429725 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:19.453995 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:19.507774 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:19.531040 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:19.541435 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:19.572432 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:19.578207 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:19.614435 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:19.650306 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 08:20:19.789033 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:19.821046 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:19.823742 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:19.846943 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:19.873107 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240501 08:20:19.882822 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:19.892777 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:19.908712 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:19.909566 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:19.931083 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:19.936702 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 08:20:20.022430 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:20.047147 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:20.048015 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:20.057247 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:20.057363 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:20.120244 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 08:20:20.208180 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:20.217255 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:20:20.265209 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:20.321287 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:20.418635 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:20.426761 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:20.439657 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:20.446158 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:20.452462 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:20.453930 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:20.456215 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:20.475550 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:20.480688 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:20.488587 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:20.525341 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:20.587829 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:20.636035 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:20.643646 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:20.658299 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:20.668880 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:20.742285 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:20.743754 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 08:20:20.850324 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:20.982457 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:21.004992 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:21.006155 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:21.033155 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:21.037351 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:21.059252 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:21.074007 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:21.073989 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:21.096170 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:21.152980 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:21.159138 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:21.290788 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:20:21.308862 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:21.314693 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:21.324102 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:21.336123 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:21.452896 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:21.487236 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:21.584070 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:21.616214 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:21.625378 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:21.664384 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240501 08:20:21.723179 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:21.730046 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:21.744421 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:21.751577 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:21.935810 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:21.941931 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:21.944782 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:21.955480 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:21.978672 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:21.988951 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:22.053879 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:22.108198 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 08:20:22.108328 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:22.119804 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:22.147639 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:22.155355 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:22.186138 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:22.304569 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:20:22.346580 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:22.354437 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:22.356870 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:22.358956 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:22.393915 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:22.418804 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:22.469573 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:22.497403 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 08:20:22.553751 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:22.556373 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:22.566686 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:22.579149 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:22.590382 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:22.613379 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 08:20:22.623418 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:20:22.639317 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:22.702628 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:22.725339 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:22.741771 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:22.909394 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 08:20:22.934935 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:22.967187 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:22.985119 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:22.986055 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:23.002475 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:23.003273 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:23.016669 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:23.021066 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:23.064067 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:23.171069 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:23.178524 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:23.183640 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:23.190940 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:23.204118 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:20:23.208087 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:23.265404 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:23.265789 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:23.469558 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:23.475188 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:20:23.476094 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 08:20:23.490181 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:23.500056 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:23.529006 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:23.539155 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 110 similar messages]
I20240501 08:20:23.608817 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:23.616678 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:23.626113 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:23.643774 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:23.657552 1256 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:23.657778 1256 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:23.658583 1256 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:20:23.661482 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:23.661679 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:23.661746 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:20:23.662134 1256 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:23.672974 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:23.714120 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:23.732713 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:23.809332 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:20:23.811630 1437 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:20:23.811815 1437 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:20:23.812570 1437 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:20:23.814265 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:20:23.821453 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:23.821620 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:20:23.823827 1576 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:23.830068 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:23.850313 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:23.851636 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:23.860630 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:23.866189 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:23.931624 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:24.080575 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:24.107091 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:24.250496 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:24.251792 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:24.261031 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:24.271780 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:24.286945 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:24.364547 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:24.409950 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:24.439347 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:24.444633 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240501 08:20:24.450381 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:24.463135 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:24.463882 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:24.561161 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 08:20:24.566606 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:24.623102 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:24.632386 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:24.637651 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:24.653225 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:24.697978 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:24.705611 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:24.714972 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:20:24.860556 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:24.874222 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:24.874881 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:24.894186 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:24.953878 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:20:24.965487 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:25.062515 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:25.067696 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:25.075613 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:25.088562 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:25.090072 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:25.138197 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:20:25.168073 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:25.225811 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:25.233245 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 08:20:25.251281 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:25.319836 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:25.377272 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:25.382639 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:25.385865 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:25.388928 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 08:20:25.444461 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:20:25.487764 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:25.502055 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:25.503422 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:25.528443 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:20:25.544371 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:25.544809 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:25.605011 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:25.613057 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:25.744529 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:25.747488 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:25.759950 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:25.766314 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:25.792202 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:25.801015 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:25.895437 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:25.998540 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 08:20:26.034229 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:26.036006 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:26.062929 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:26.148934 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:26.163558 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:26.163962 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:26.164917 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:20:26.175024 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:26.221215 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:26.290099 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:26.331755 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:26.347843 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:26.349411 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:20:26.365327 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:26.372249 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:26.389035 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:26.444072 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:26.528159 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:26.547868 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:20:26.626828 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:26.789813 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:26.791884 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:26.799118 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:26.813951 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240501 08:20:26.826423 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:26.844169 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:26.878423 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:26.928273 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:26.941776 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:26.993839 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:26.997651 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:27.021021 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:27.021121 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240501 08:20:27.030474 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:27.097700 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:27.115190 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:27.138653 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:27.198091 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:27.234418 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:27.237112 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:27.327076 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:20:27.404045 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:27.409504 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:27.426347 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:27.440586 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:27.463006 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:27.467350 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:27.560976 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:27.571317 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 08:20:27.579213 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:27.609781 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:20:27.618314 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:27.619272 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:27.629801 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:27.637334 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:27.679239 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:27.727484 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:27.789801 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:27.811074 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:27.877071 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:27.894506 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:27.919862 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 08:20:27.955304 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 08:20:27.977684 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:28.043951 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:28.045971 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:28.056172 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:28.057410 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:28.064328 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:28.085922 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:28.086822 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:28.089836 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:20:28.248960 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:28.251983 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:28.252182 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:28.268438 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:28.320444 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:28.407502 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:28.448807 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:28.495092 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:20:28.611374 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240501 08:20:28.612989 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:28.627182 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:20:28.648568 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:28.653810 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:28.672124 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:28.688385 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:20:28.689740 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:28.692485 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:28.770787 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:28.776868 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240501 08:20:28.885483 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:28.887037 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:28.899204 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:28.912175 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:28.913020 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:28.937597 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:28.969486 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:28.982970 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:29.044018 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:29.058316 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:20:29.269593 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:29.279299 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:20:29.298508 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:29.298692 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:29.309880 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:29.316236 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:29.341655 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 08:20:29.390383 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:29.394779 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:29.460198 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:29.469133 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240501 08:20:29.475250 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:29.491962 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:29.493222 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:29.499192 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:29.500422 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:29.612717 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:29.721400 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:29.745008 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:29.753880 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:29.787472 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:29.902396 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:20:29.911657 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:29.917779 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:29.926126 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:29.934901 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:20:29.945674 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:30.008949 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:20:30.021257 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 08:20:30.036566 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:30.082298 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:20:30.120735 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:30.130393 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:30.139869 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:30.140887 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:30.155118 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:30.169615 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:30.178398 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:30.245832 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:30.284955 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 08:20:30.389201 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:30.401002 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:30.429829 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:30.430822 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:30.461730 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:30.497347 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:30.518376 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:30.557818 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:30.564777 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:30.573518 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:30.577404 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:30.627789 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:30.667888 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:30.755081 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:30.757468 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:30.777352 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:30.779266 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:30.895663 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:31.038957 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:31.051491 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:31.062639 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:31.153913 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:20:31.164228 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:31.177420 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:31.185945 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:31.195482 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:31.223179 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:31.326007 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:31.368500 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:31.369560 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:31.384858 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:31.389355 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:31.440454 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:31.448685 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:31.452514 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 08:20:31.478127 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:31.553943 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:31.591017 1256 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:31.591199 1256 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:31.591747 1256 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:20:31.593061 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:31.593256 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:31.593330 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:20:31.593605 1256 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:31.645328 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:31.677935 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:20:31.684314 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:31.784725 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:31.788219 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:31.788949 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:31.809747 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:31.843866 6484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240501 08:20:31.851653 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:20:31.852105 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 08:20:31.872162 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:31.872601 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 08:20:31.914768 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:31.946233 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:31.970613 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:20:32.016817 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:32.023707 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:32.030719 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:32.040824 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:32.045715 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:32.065644 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:32.083302 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:32.146579 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:32.261205 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:32.268658 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:32.280781 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:20:32.429098 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:32.437106 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:32.441686 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:32.450084 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:32.455437 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:32.507360 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:32.580633 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 08:20:32.595553 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:32.603518 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:32.628075 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:32.632194 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:32.639662 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:20:32.643996 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:32.646035 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:32.653614 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:32.663516 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:32.670379 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 08:20:32.703178 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:32.737150 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:32.764029 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:32.793426 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:32.823292 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:32.860760 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:32.939723 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 08:20:33.051286 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:33.074748 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:33.076166 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:33.081041 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:33.085009 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:33.167258 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:33.180341 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:33.216243 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:33.295253 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:33.297566 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:33.315016 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:33.325515 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:33.355777 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d7903106d7e74e1ea102b1abdbfce63e: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:20:33.521591 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:20:33.529866 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:33.629974 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 08:20:33.692134 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:33.697188 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:20:33.707321 2715 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:20:33.707566 2715 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:20:33.708413 2715 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
I20240501 08:20:33.710003 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:33.712191 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:20:33.712430 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:33.712529 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:20:33.713531 2715 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:33.727368 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:20:33.730391 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:33.745988 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:33.746443 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:33.747524 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:33.753010 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:33.761833 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:20:33.939404 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:33.941645 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:33.943161 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:33.955333 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:33.961771 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 08:20:33.972309 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:34.023262 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:34.036046 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:34.046510 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:34.204221 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:34.325294 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:34.354286 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:20:34.355448 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:34.372558 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:34.373572 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:34.396004 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:34.410779 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:34.451114 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 08:20:34.547591 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 08:20:34.575393 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:34.588611 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:34.591339 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:34.601747 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:34.623414 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:34.624462 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:34.690531 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:34.747370 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:34.809360 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:34.811470 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:34.849825 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:34.881909 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 193de3c11ae54408a64ae3007bf88920: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:34.889242 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:34.900244 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:20:34.956967 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:20:35.018021 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:35.028985 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:35.029556 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:35.030078 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:35.046772 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 08:20:35.104656 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 08:20:35.107525 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:20:35.108464 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:35.117038 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:35.121469 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:35.161746 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:35.172695 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:35.190081 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:35.212967 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:20:35.221951 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:35.230841 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:35.245301 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:35.250732 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:35.287544 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:35.319288 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:35.401005 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:35.485018 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:35.586896 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:35.639827 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:35.651722 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:35.652637 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:35.659271 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:35.664011 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:35.673354 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:35.740170 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:35.777931 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:20:35.865777 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:35.870590 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:35.876724 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:35.878701 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:36.076896 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:36.105476 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:36.206555 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:36.223896 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240501 08:20:36.277879 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:36.290810 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:36.296283 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:20:36.304370 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:36.305079 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:36.315130 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:36.334479 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:36.359553 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:36.384193 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:20:36.487977 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:36.490022 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:36.494563 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:20:36.495432 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:36.502547 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:36.524857 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:36.560943 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:36.603506 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:36.861311 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:36.890432 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:36.899226 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:36.906219 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:36.911280 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:36.960520 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:37.013216 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:37.069847 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:37.073218 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 08:20:37.079901 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:37.109020 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:20:37.111894 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:37.130620 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:37.135769 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:37.147825 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:37.157821 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 08:20:37.200799 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:20:37.256531 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:37.321734 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:37.384567 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:37.395292 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:37.399160 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:37.417001 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:37.419565 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:37.499891 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240501 08:20:37.519601 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:37.529155 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:37.558795 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:20:37.561939 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:37.575035 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:37.591953 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:37.629357 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:37.631733 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:37.687820 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 08:20:37.747123 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:37.754925 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:20:37.764987 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:37.769390 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:37.770778 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:37.782229 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:37.789748 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:37.805537 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:37.819485 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:37.837958 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:37.861696 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:37.970381 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:37.995493 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 08:20:37.996373 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:38.179287 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:38.184653 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:38.197242 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:38.219461 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:38.221639 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:38.402537 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:38.404140 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:38.413367 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:38.413425 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:38.436031 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:38.534282 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:38.550808 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:20:38.636384 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 08:20:38.723762 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:38.769228 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:20:38.801841 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:38.807485 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:38.828534 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:38.838515 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:38.888960 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:38.901180 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:38.922698 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:38.965541 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:38.968186 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 08:20:38.977789 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:38.979226 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:38.996140 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:20:39.011301 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:39.022894 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:39.038475 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:39.044133 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:39.111373 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:39.250982 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:20:39.374136 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:39.417361 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:39.421854 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:39.427474 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:39.435518 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240501 08:20:39.444684 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:39.507053 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 08:20:39.579674 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:39.604516 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:39.630020 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:39.649453 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:39.649915 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:20:39.654115 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:39.655058 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:20:39.662688 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:39.693158 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:20:39.707419 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:39.834936 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:39.857568 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:39.874142 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:39.930231 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 08:20:39.998704 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:40.031404 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:20:40.109977 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:40.112612 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:40.118017 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:40.118158 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:20:40.135846 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:40.146593 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:40.162547 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:40.174878 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:20:40.175864 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:40.186957 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 08:20:40.198665 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:40.214470 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:40.217231 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:40.221446 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:40.243026 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:40.250777 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:40.278756 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:20:40.307475 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:40.316152 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:40.322484 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:40.324605 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:40.348840 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:40.413070 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:40.414053 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:40.622668 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:40.754463 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:40.775350 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:40.782058 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:40.801473 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:40.835884 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:40.855322 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:40.976325 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:40.977963 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:40.991621 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:41.005465 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:41.042382 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:41.176164 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:41.207239 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:41.224419 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:41.337244 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:41.355686 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:41.386741 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:41.396083 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:41.399787 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:41.417281 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:41.423758 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:41.423892 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:20:41.448860 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:20:41.593791 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:41.594344 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:41.596033 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:41.602582 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:41.603724 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:20:41.605957 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:41.675699 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:41.743853 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:41.771495 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 08:20:41.920588 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:41.985221 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:41.986135 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:42.006808 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:42.008455 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:20:42.015038 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:42.024183 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 08:20:42.048110 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:42.078522 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:42.091077 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:42.133352 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:42.149855 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:42.155462 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240501 08:20:42.183089 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:42.191642 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:42.192039 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:42.203605 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:42.293833 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:42.358064 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:42.374987 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 08:20:42.496775 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:42.519292 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:42.536211 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:42.540108 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:42.576463 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:42.599954 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240501 08:20:42.609979 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:42.626093 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:42.627095 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:42.628966 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:42.634485 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:42.711007 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:42.730913 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:42.812415 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:42.830349 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:20:42.834389 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:42.834422 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:42.839759 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:20:42.842953 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:42.844902 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:42.852277 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:20:42.859242 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:42.868945 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:42.885594 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:42.928503 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:43.034333 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:43.047019 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 08:20:43.047875 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:43.263659 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:43.265105 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:43.275842 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:43.286582 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:43.461685 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:43.468135 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:43.475795 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:43.475798 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:43.497572 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:43.519198 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:43.520144 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:43.544818 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:43.558099 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:43.610467 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:43.687426 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 08:20:43.695749 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:20:43.759454 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:43.813375 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:20:43.859817 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:43.866945 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:43.869822 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:43.908221 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:43.923233 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:43.932933 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:44.049469 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:44.080462 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:44.083822 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:44.091941 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:44.109584 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:20:44.112813 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:44.128289 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:44.195680 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:44.304217 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 08:20:44.473939 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:20:44.492906 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:44.512174 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:44.516428 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:44.516855 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:44.518826 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:44.524212 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:44.529639 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:44.548739 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:44.552942 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:44.586540 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:44.635195 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:20:44.640805 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240501 08:20:44.717623 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:44.747581 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:44.751700 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:44.756278 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:44.809779 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 08:20:44.829317 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:44.830353 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:44.953408 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:44.960052 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:45.032724 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:45.080435 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:45.102097 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 08:20:45.116972 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:45.118667 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:20:45.157938 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:45.158646 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:45.164861 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:45.169772 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:45.232939 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:20:45.254501 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:45.281204 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:45.317373 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:45.330888 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:20:45.367729 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:45.372851 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240501 08:20:45.379004 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:45.385962 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:45.399001 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:45.407802 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:45.415438 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:45.449448 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:45.466313 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:45.484050 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:45.526733 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:45.539268 3420 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:45.539453 3420 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:45.540113 3420 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:20:45.541728 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:45.541924 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:45.541997 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:20:45.542217 3420 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:45.645016 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:45.676162 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:45.794484 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:45.803464 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:45.805655 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:45.817474 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:45.953958 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:45.974009 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:20:46.017283 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:46.027568 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:46.029004 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:46.046748 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:46.058717 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:46.070780 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:46.187640 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:46.241662 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:46.332952 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:20:46.411629 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:46.413276 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:46.435920 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:46.438509 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:46.455785 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:46.568753 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:46.616994 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:46.632339 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:46.634028 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:46.642076 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:46.663105 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:46.684674 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:46.729208 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:46.732650 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:46.829787 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 08:20:46.885738 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:47.004926 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:47.013985 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:47.027158 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:47.044898 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:20:47.048046 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:47.050765 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:47.050941 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:47.065495 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:47.072687 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:47.113819 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:20:47.123138 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 08:20:47.180305 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:47.200701 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:47.200742 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 08:20:47.243233 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:20:47.255705 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:47.256121 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:47.270066 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:47.273201 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:47.382858 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:47.450908 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:47.499179 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:47.524720 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:47.582763 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:47.638830 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:47.651448 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:47.667841 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:47.668270 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:47.723554 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:47.797418 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:47.797456 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:47.836566 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 08:20:47.847715 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:47.850356 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:47.855978 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:47.858464 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:47.867307 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:47.884879 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:47.897184 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:47.917713 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:47.940192 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:47.982493 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 08:20:48.014212 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:48.040681 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:48.051829 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 08:20:48.067000 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:48.119793 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:48.241859 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:48.283205 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:20:48.286688 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:48.297853 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:48.308183 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:48.310493 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:48.434902 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:48.469045 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:48.496203 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:48.522754 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:48.525095 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:48.561275 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:48.605073 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:48.630077 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:48.692154 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:48.712949 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240501 08:20:48.713985 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:48.719200 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:20:48.911742 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:20:48.939802 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:48.942834 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:48.955255 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:48.958391 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:49.045737 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:49.048492 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:49.065773 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:49.068971 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 08:20:49.123252 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:49.123800 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:49.139530 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:49.148751 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:49.150223 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:49.169076 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:49.173909 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:49.431892 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:49.437846 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:49.530187 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:49.554881 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:49.567862 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:49.570600 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:49.574236 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:49.580825 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:49.663487 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 08:20:49.685786 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:49.686542 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:49.740355 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240501 08:20:49.750748 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:49.770125 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:49.770184 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:49.770565 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:49.774525 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:20:49.788082 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:49.815861 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:49.871481 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:49.979403 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:50.002969 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:50.063021 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:50.165760 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:50.172883 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:50.173772 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:50.176400 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:20:50.187011 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:50.190366 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:50.243032 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 08:20:50.243986 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:50.267683 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 08:20:50.284741 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:20:50.358032 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:50.373283 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:50.373456 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:50.376037 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:50.377734 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:50.408941 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:50.425745 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:50.519268 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:50.597561 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:50.653764 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:50.709195 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:20:50.718888 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:50.742616 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:20:50.777776 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:50.783499 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:50.790302 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:50.790449 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:50.798774 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:20:50.820225 4256 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:20:50.820416 4256 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:20:50.821097 4256 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:20:50.826996 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:20:50.842541 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:50.842737 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:20:50.843040 4256 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:50.845080 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 08:20:50.945995 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:50.975694 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:20:50.981304 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:50.989035 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:51.039726 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:20:51.044616 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:51.053095 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:51.100849 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:51.117226 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:51.163471 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:51.240914 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:51.323973 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:51.425805 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:20:51.484735 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:51.505597 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:51.507030 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:51.511168 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:51.542562 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:51.548552 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:20:51.575182 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 08:20:51.682112 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:51.708648 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:51.721581 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:20:51.731865 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:51.735035 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:51.736387 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:51.781324 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:51.881798 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:51.937041 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:20:52.034638 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 08:20:52.111129 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:52.112833 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:52.117908 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:52.130733 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:52.149752 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:52.153779 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:52.161206 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:20:52.200989 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:20:52.239761 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 08:20:52.255170 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:52.261435 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:20:52.296608 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:52.297097 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:20:52.324476 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:52.330392 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:52.336119 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:52.339470 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:52.341265 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:52.394986 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:52.513350 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:52.604218 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:20:52.614188 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 08:20:52.655270 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:20:52.668437 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:20:52.685286 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:52.733477 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:52.744310 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:52.751716 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:52.753072 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:52.761092 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:52.798202 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:20:52.934517 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:52.944743 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:52.948374 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:52.950234 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:52.962641 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:52.977778 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:53.003149 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:20:53.011587 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:53.098140 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 08:20:53.160045 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:53.225420 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:53.314932 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:20:53.328150 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:53.341744 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:53.348070 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:53.369316 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:53.382298 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:20:53.409898 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:20:53.410864 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 08:20:53.460050 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:53.482661 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:53.542243 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:20:53.570762 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:53.579159 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:53.595891 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:20:53.604668 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:53.619323 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:53.632449 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:53.668753 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:53.729056 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 08:20:53.790284 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:20:53.872327 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:53.947144 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 08:20:54.014483 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:20:54.019433 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:54.032987 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:54.035190 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:54.041255 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:54.059733 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:54.094856 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:20:54.192924 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:54.232092 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:54.234486 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:54.262123 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:54.269246 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:54.333652 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:54.425817 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:54.427173 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:20:54.469187 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:54.481483 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:20:54.583930 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 08:20:54.634358 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:20:54.646675 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:54.650557 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:54.660677 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:54.661224 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:54.666132 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:54.744916 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 08:20:54.784685 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:20:54.786387 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:20:54.846601 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:54.857605 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:54.858850 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:54.859059 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:54.928583 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:54.940886 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:20:54.942585 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:54.961458 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:20:54.961894 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:54.977191 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:54.984972 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:20:54.999791 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:20:55.033529 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:55.084823 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:20:55.087256 4876 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:20:55.087427 4876 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:20:55.088002 4876 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:20:55.089180 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:55.089329 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:55.089407 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:20:55.089632 4876 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:55.130332 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 08:20:55.136870 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:20:55.239766 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:55.255991 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:55.263031 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:55.279096 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:55.315016 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:20:55.319929 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:55.385880 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:55.419234 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:55.432178 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:20:55.432888 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:55.441282 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:55.460577 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:55.473655 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:55.490799 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:55.559531 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:55.639250 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:55.640137 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240501 08:20:55.856855 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:55.857338 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:55.861008 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:55.867046 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:55.899607 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:20:55.917714 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:55.926767 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:55.991793 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:56.014463 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:20:56.045120 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:56.049959 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:56.061530 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:20:56.065225 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:56.069903 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:56.085240 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:56.085994 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:56.110093 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:56.307755 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:56.379523 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 08:20:56.489615 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:56.509438 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:56.514178 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:20:56.531037 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:56.538185 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:56.543639 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:56.548131 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:56.722453 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:56.732147 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:56.734982 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:20:56.737036 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:56.743924 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:56.752125 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:56.754817 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:56.854444 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:20:56.908372 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:20:57.032064 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:57.035383 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:57.126132 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:57.145166 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:57.149040 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:20:57.161386 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:57.172030 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:57.174170 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 08:20:57.231261 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:20:57.247663 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 08:20:57.296891 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:57.320147 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:20:57.322314 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:57.337605 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:57.360486 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:57.361507 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:57.411132 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:20:57.448199 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:57.461663 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:57.486948 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:57.546837 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:20:57.559270 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:57.566010 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:20:57.650913 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:57.653259 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:20:57.669826 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:57.706828 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 08:20:57.744041 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:20:57.744562 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:57.745352 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:57.747711 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:57.770174 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:57.831748 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:20:57.888186 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:57.922821 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:57.931901 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240501 08:20:57.962667 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:57.967895 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:57.972463 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:57.977115 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:20:57.977775 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:57.990747 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:58.104943 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 08:20:58.105825 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:58.126657 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:58.320518 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:20:58.366850 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:58.375954 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:58.378441 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:58.380051 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:58.433120 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:58.449330 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:20:58.502491 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:58.503341 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:58.530678 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:58.572301 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:58.584075 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:58.588487 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:20:58.590982 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:58.617376 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:58.665143 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:58.723623 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:20:58.781838 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 08:20:58.782864 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:58.817301 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:20:58.818135 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240501 08:20:59.010502 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:59.010527 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:59.022042 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:59.028432 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:59.032063 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:59.088300 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:59.226006 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:59.229841 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:59.233120 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:59.238605 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:59.247264 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:59.253662 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240501 08:20:59.258549 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:20:59.262365 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:20:59.354624 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:20:59.442246 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:20:59.463925 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:59.472209 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:20:59.499392 4256 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:20:59.499594 4256 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:20:59.500273 4256 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:20:59.501641 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:20:59.501782 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:20:59.501909 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:20:59.502151 4256 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:20:59.587422 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240501 08:20:59.636343 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:59.639384 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:59.649199 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:20:59.659514 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:59.660331 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:59.692644 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:20:59.726915 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:20:59.846127 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:20:59.846809 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:20:59.861295 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:20:59.867604 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:20:59.876009 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:20:59.879119 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:20:59.904842 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:20:59.915904 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:21:00.021776 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:00.107369 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:00.140108 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:21:00.189002 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:21:00.191797 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 08:21:00.195762 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:00.248495 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:00.270226 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:00.284541 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:00.297786 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:00.307897 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:00.316303 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:00.328023 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:21:00.329133 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:00.335803 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:00.338068 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:00.385746 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:21:00.481590 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:00.482985 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:21:00.487521 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:00.491698 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:21:00.520988 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:00.521493 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:00.546505 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:00.639847 5821 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:21:00.640058 5821 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:21:00.640760 5821 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:21:00.642635 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:00.642822 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:00.642918 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:21:00.643177 5821 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:21:00.669586 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 08:21:00.772854 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:00.869937 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:00.877074 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:21:00.898144 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:00.917821 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:00.947881 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:00.949355 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:00.958896 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:00.978211 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:21:00.982205 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:01.037634 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:01.040046 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:21:01.140396 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:21:01.154578 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:01.154623 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:01.202311 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:01.202387 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:01.203420 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:01.320530 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 08:21:01.384781 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:01.468364 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:21:01.481966 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:01.590864 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:01.591300 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:01.605674 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:01.611939 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:01.655934 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 08:21:01.663460 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:01.820070 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:01.827819 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:01.836572 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:01.837757 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:21:01.838388 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:01.843147 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:01.900681 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:21:01.913715 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:01.957756 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:21:01.964138 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:02.093034 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:02.124477 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:02.177081 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 08:21:02.213989 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:02.235289 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:21:02.241402 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:02.247529 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:02.254904 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:02.267555 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:02.340763 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:02.344853 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:21:02.454573 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:02.459558 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:02.469151 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:02.470571 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:02.487883 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:02.521230 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:02.560360 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:02.602924 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:02.683890 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 08:21:02.713586 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:02.754851 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:21:02.824535 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:02.834468 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:02.857717 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:02.869014 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:02.879473 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:02.880044 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:02.881053 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:02.892333 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:21:02.931051 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:02.949440 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:02.994278 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:03.029554 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:03.038498 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:21:03.064229 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:03.074069 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:03.083715 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:03.087376 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:03.134124 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 08:21:03.136049 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:03.136505 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:03.145795 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 08:21:03.394016 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:03.448449 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:21:03.481966 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:03.483351 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:03.483814 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:03.488387 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240501 08:21:03.496332 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:03.516144 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:03.534523 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:03.536000 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:21:03.634938 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:21:03.690834 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:03.716610 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:03.720119 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:03.726522 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:03.793671 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:21:03.800256 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:03.804783 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240501 08:21:03.805570 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:21:03.852514 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:03.896301 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:21:03.906718 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:03.946616 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:04.117921 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:04.137706 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:04.150049 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:04.177811 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:04.193019 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:04.316341 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:21:04.365782 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:04.366222 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:04.387988 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240501 08:21:04.390460 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:04.393229 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:04.428333 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:04.435322 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:04.535076 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:04.557652 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:04.585094 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:04.597867 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:21:04.642203 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:04.650606 6484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240501 08:21:04.651708 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:21:04.754575 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:04.829728 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 08:21:04.846246 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:04.857687 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:04.861215 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:04.875617 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:04.912758 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:04.915020 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:04.977566 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:05.002171 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:05.033854 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:05.038743 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:21:05.052737 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:05.061276 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:05.064090 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:05.081270 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:05.127396 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:05.136024 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:05.234545 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:05.270370 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:05.272120 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:05.305364 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:05.330652 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:05.384310 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:05.386942 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:21:05.428944 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:21:05.462606 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:05.465474 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240501 08:21:05.484303 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:05.486570 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:05.489758 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:05.518013 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 08:21:05.537688 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:05.544515 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:05.546698 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:05.576488 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:05.635072 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:21:05.675047 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:05.703161 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:05.711113 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:05.714475 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:05.915997 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:05.921703 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:05.941340 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:06.047089 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:06.125012 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:06.131357 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:06.134176 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:06.138355 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:06.147092 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:06.164043 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:06.191394 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d7903106d7e74e1ea102b1abdbfce63e: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:21:06.232422 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:06.312433 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 08:21:06.327831 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:06.335731 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:06.346626 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:06.358368 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:06.368227 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:06.391801 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:06.437754 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:06.580829 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:06.590215 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:21:06.758118 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:06.765036 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:06.794773 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:06.802604 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:06.818086 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:06.872042 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 08:21:06.916646 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:06.946055 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:06.957290 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:07.017716 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:21:07.023687 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:07.039705 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:07.040485 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:07.041467 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:07.152701 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:21:07.272903 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:07.285367 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:07.327261 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 08:21:07.345700 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:07.393940 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:07.433459 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:21:07.445248 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:07.455622 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:07.462502 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:21:07.464588 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:07.467451 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:07.483080 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:07.566371 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:07.600526 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:21:07.663331 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:07.675509 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:07.683197 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:21:07.683754 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:07.690634 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:21:07.691771 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:07.715965 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:07.726490 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 193de3c11ae54408a64ae3007bf88920: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:07.818814 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:07.833076 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:07.835916 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:07.907898 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:21:07.912546 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:08.023623 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:08.030681 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:21:08.040815 6180 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:21:08.041013 6180 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:21:08.041682 6180 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:21:08.043998 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:21:08.045392 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:08.045584 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:08.045684 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:21:08.046012 6180 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:21:08.056983 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:08.089579 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:08.093881 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:08.101755 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 08:21:08.101934 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:08.113088 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:08.144776 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 08:21:08.169893 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:08.175958 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:08.212366 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 08:21:08.296365 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:08.296365 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:21:08.298961 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:08.307317 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:08.325069 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:08.353310 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:21:08.356081 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:08.556493 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:08.608506 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:21:08.644471 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:08.648787 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:21:08.669461 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:21:08.696288 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240501 08:21:08.708077 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:08.739017 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:08.742697 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:08.749109 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:08.817819 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 08:21:08.862475 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:08.905514 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:08.907389 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 67 similar messages]
I20240501 08:21:08.957291 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:08.958099 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:08.970407 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:08.971405 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:09.011173 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:09.107327 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:09.349404 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:21:09.363772 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:09.366490 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:09.377787 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:09.378834 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:09.410053 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:09.423609 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:09.489832 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:09.496491 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:21:09.573597 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:09.575599 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:09.587332 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:09.592545 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:09.609115 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:09.646354 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 08:21:09.666895 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:21:09.733852 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:09.734555 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 08:21:09.824669 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:09.841861 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:09.845404 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:09.908934 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:09.930496 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:09.971339 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:09.972640 7434 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:21:09.972817 7434 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:21:09.973515 7434 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:21:09.975970 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:09.976164 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:09.976244 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:21:09.976493 7434 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:21:10.004765 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:10.004961 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:21:10.008035 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:10.016340 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:10.023653 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:10.049448 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:10.050639 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:10.115378 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:10.166029 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:21:10.196893 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:10.205554 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:10.209021 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:10.214244 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:10.241778 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:10.283807 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:10.316429 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:10.330540 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:10.386796 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:10.389533 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:10.408038 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 08:21:10.411873 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:10.461463 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 08:21:10.476213 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:21:10.609912 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:10.612581 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:21:10.619016 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:10.628715 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:10.632462 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:10.718463 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:10.787689 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:21:10.804005 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:10.815411 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:10.825027 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240501 08:21:10.828331 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:10.846475 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:10.925519 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:10.936264 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:11.071691 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:11.141641 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:11.153182 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:11.158311 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:11.189289 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:21:11.248680 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:11.254523 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:11.259187 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:11.260071 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:11.280586 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:11.353894 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:11.356741 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:11.377709 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:11.418174 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240501 08:21:11.451000 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:11.460880 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:11.477046 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:11.479660 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:11.765534 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:11.767385 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:11.779352 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 08:21:11.817839 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:11.818557 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:11.838919 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:11.867643 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:11.873942 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:11.881837 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:11.895499 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:11.927642 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:21:12.085204 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:12.096058 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:12.099253 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:12.127039 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:12.189096 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 08:21:12.208927 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:12.312171 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:12.375023 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:12.376259 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:12.382948 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:12.387815 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:12.394402 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:12.416509 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:12.427155 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:21:12.462006 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:21:12.509831 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:12.528298 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:12.535198 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:12.535383 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:12.572306 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:12.572417 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:12.661542 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:12.688620 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:12.719604 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:12.721122 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:21:12.752676 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:12.762532 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240501 08:21:12.766458 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:12.767692 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:12.769420 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:12.771291 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:12.813354 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:12.861575 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:12.982102 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:13.080415 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:13.097314 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:13.121245 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:13.131139 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:21:13.174078 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:13.174862 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:13.178985 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:13.187928 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:13.200264 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:13.270992 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240501 08:21:13.276834 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240501 08:21:13.395246 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:13.399374 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:13.406652 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:13.424206 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:13.472698 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:13.473892 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:13.496073 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:13.496224 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:13.558499 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:13.587751 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:13.718375 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:13.740489 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:13.820428 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:13.821841 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:13.827350 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:13.830667 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:13.890285 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:13.907862 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:13.945235 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 08:21:13.962450 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:21:14.011901 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:14.014062 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:14.021796 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:14.028805 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:21:14.029019 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:14.108027 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 08:21:14.182725 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:14.304334 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:14.311954 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:14.353806 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:14.402532 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:21:14.420862 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:14.422825 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:14.432101 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:14.434370 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:14.483937 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 08:21:14.511108 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:14.609648 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:14.628118 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:14.630883 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:14.631866 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:21:14.633857 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:14.660748 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:14.679641 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:21:14.750313 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:14.775121 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:14.811570 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 08:21:14.816421 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:14.868041 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:14.948359 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:14.962649 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:15.064319 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:21:15.064553 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:15.069082 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:21:15.070003 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:15.070020 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:15.076743 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:15.114684 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:15.125324 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:15.154837 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:21:15.255327 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:15.256698 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:15.262974 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 08:21:15.263258 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:15.282963 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:15.283185 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:15.328701 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:15.364219 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:15.366838 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:15.367468 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:15.373649 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:15.460662 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:15.531126 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:15.543895 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:21:15.544728 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:15.563005 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:21:15.692036 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:15.696260 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:15.713670 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:15.719115 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:15.773677 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 08:21:15.796191 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:15.820268 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:15.848058 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:15.882196 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:21:15.882735 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:15.895735 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:15.909533 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:15.909718 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:16.035501 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:16.039446 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:16.171156 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:16.271767 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:16.286216 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:16.297223 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:16.298899 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:16.299686 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:16.301416 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:16.319424 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:16.473603 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:16.475262 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:16.482836 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:16.484455 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:16.506387 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:16.529906 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:16.636160 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 08:21:16.757336 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:16.769658 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:16.774371 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:16.915359 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:16.919610 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:16.920156 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240501 08:21:16.926218 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:16.929987 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:16.982107 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:16.996512 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:21:17.090294 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:17.101298 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:17.106325 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:17.110597 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:21:17.111553 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:17.146216 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:21:17.153512 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:17.159565 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:17.196768 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:17.243960 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:17.284138 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:17.292652 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:17.369190 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:17.401239 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 08:21:17.570758 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:17.571461 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:17.580547 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240501 08:21:17.620282 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:17.624747 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:17.630064 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:17.630733 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:17.673770 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:17.694094 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:17.695312 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:17.700264 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:17.742328 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:17.803171 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:17.813854 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:17.818212 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:21:17.853432 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:17.868574 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:17.869218 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:21:17.880357 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:17.883200 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:17.942705 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:17.993052 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:18.108853 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:18.243619 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:18.269054 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:18.273481 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:18.274797 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:18.275750 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 08:21:18.276857 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:18.288651 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:18.292430 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:18.321712 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 08:21:18.397035 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:18.420539 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:18.428602 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:21:18.471304 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:18.474061 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:18.477679 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:18.484671 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:18.494601 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:18.695605 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:18.741528 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:18.796854 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:18.863812 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:21:18.871802 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:18.884419 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:18.887109 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:18.890717 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:18.968063 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 08:21:19.041121 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:21:19.070485 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:19.072949 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:19.078037 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:19.084390 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:19.105743 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:19.110062 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:19.116124 8527 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:21:19.116366 8527 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:21:19.117142 8527 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:21:19.120801 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:19.121186 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:21:19.121315 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:21:19.121959 8527 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:21:19.133285 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:19.158468 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:19.160234 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 08:21:19.171824 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 08:21:19.199095 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:19.361451 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:19.379909 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:21:19.481663 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:19.483163 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:19.508816 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:21:19.509343 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:19.525624 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:19.536434 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:19.584401 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:19.633597 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:19.656036 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:21:19.657781 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:19.699314 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:19.726855 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:19.729938 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:19.736564 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:19.741771 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:19.747094 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:19.763866 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:19.946913 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:19.960207 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:19.982340 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:20.010210 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 08:21:20.024566 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:20.055133 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:20.057528 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 08:21:20.110491 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:20.155745 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:20.163026 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:20.169358 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:20.178627 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:20.184072 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:20.289780 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:20.329192 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:20.337587 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:21:20.338465 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:20.340584 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:20.361296 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:20.362951 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:20.425328 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:20.430397 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:20.495132 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:20.547667 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:20.574236 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:20.593839 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:21:20.710427 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:20.718727 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:20.764628 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:20.766355 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:20.776160 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:20.780902 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:20.849659 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 08:21:20.853713 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:20.920955 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:21:20.951138 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:20.959694 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:20.962620 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:20.962764 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:20.993036 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:21.069703 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:21.175786 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:21.207252 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:21.261190 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:21.339177 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:21:21.348654 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:21.366240 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:21.366864 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:21.386525 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:21.392017 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:21.557673 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:21.562970 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:21.578401 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:21.612551 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:21.650885 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:21.652473 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:21.664680 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 08:21:21.666226 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 08:21:21.729868 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:21.737387 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:21.790725 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:21.859822 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:22.007805 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:22.010336 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:22.012025 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:22.025667 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:22.054087 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:21:22.054411 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:22.058509 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:22.068852 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:22.110324 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:22.137228 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:22.180651 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:21:22.193689 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:22.203516 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:22.211668 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:22.211931 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:21:22.216962 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:22.269940 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:22.329793 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:22.356624 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:22.453341 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:22.463855 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 08:21:22.469388 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:22.598632 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:21:22.623549 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:22.643107 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:22.645865 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:22.647651 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:22.666482 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 08:21:22.667217 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:22.681697 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:22.695637 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:22.717244 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:22.798894 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:21:22.850574 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:22.857578 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:22.860589 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:22.866380 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:22.870760 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:22.923462 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:22.975173 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:23.005321 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:23.139508 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:23.210522 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:23.255703 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:23.285699 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:23.285718 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 08:21:23.286909 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:23.293514 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:23.294467 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:23.302556 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:23.348628 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 08:21:23.439846 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:21:23.479540 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:23.490051 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:23.501072 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:23.503580 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:23.583339 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:23.610491 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:23.674845 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:23.707101 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:23.749344 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:23.913851 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:23.924552 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:23.926601 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:21:23.947749 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:23.952247 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:23.964493 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 08:21:23.981336 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 08:21:24.018229 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:24.072654 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:21:24.073567 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 08:21:24.114161 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:21:24.154479 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:24.162509 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:24.167169 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:24.173836 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:24.173892 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:24.174031 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:24.193149 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:24.228783 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:24.255049 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:24.503389 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:24.503564 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:24.539341 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:21:24.579512 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:24.580482 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:24.589176 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:24.591441 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:24.600562 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:24.608011 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:24.633486 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:21:24.664469 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:24.741307 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:24.747581 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:24.795686 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:24.796243 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:24.803881 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:24.808341 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:24.809342 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:24.886054 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:24.902271 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:24.946525 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:24.969417 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:25.014052 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:25.041596 9229 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:21:25.041791 9229 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:21:25.042511 9229 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:21:25.044046 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:25.044190 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:25.044272 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:21:25.044499 9229 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:21:25.049237 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 08:21:25.052655 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 08:21:25.081758 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:25.213882 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:21:25.224961 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:25.242544 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:25.247478 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:25.248044 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:25.260823 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:25.276532 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:25.296861 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:25.421002 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:25.424887 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:21:25.432415 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:25.449138 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:25.454046 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:25.466539 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:25.626495 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:21:25.627316 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:25.693233 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:25.697419 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:25.797542 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:25.829594 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:25.859244 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:25.860517 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:25.865741 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:25.871929 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:26.019026 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:26.055903 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:21:26.063206 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:26.072412 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240501 08:21:26.074855 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:26.080399 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:26.088359 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:26.111532 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:26.137493 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:26.372203 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:26.394642 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:26.497552 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:26.502424 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:26.508085 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:26.524734 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:26.579308 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:26.610715 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:26.654978 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:26.677279 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 08:21:26.682718 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:21:26.709897 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:26.719836 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:26.721550 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:26.725456 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240501 08:21:26.732445 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:26.813122 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:26.833485 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:26.899546 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:26.908795 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:26.932886 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:21:26.948262 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:26.955013 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:26.995335 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:27.127231 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:27.129267 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:27.130019 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:27.139065 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:21:27.146653 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:27.150882 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:27.176915 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 08:21:27.216456 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:27.256609 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:27.266428 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:21:27.301172 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:27.309082 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:27.309568 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:27.310231 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:21:27.319708 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:27.326856 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:27.383067 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:27.390623 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:27.436270 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:27.437767 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:27.439896 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 08:21:27.513984 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:27.562464 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:21:27.715735 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:27.731375 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:27.747949 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:27.758735 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:27.765018 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:27.812268 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:27.824374 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:27.829030 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:27.866303 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:21:27.961050 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:27.971714 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:27.976166 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:27.986178 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:28.000830 9382 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:21:28.001066 9382 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:21:28.001777 9382 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:21:28.021205 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:28.021502 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:28.021783 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:28.022136 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:21:28.022572 9689 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:21:28.316507 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 08:21:28.317472 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:28.344862 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:28.381399 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:28.384179 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:28.384977 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:28.407019 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:28.425014 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240501 08:21:28.426640 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:28.435889 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:28.435935 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:28.443792 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:28.519205 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:21:28.616647 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:28.636814 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:28.638017 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:28.662788 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:28.749154 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:28.751665 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:28.843894 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:28.862607 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:28.935416 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:28.941082 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:28.996528 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:29.014617 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 114 similar messages]
I20240501 08:21:29.064307 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:29.078716 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:29.087801 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:29.090863 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:29.170789 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:21:29.277822 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:21:29.278937 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:29.285187 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:29.285945 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:29.289714 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240501 08:21:29.294086 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:29.301452 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:29.308352 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:21:29.339128 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 08:21:29.350069 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 08:21:29.383100 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:29.398746 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:29.436983 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:29.535051 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:29.579612 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:29.597304 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:29.600522 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:21:29.691006 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:29.713486 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:29.714381 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:29.718089 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:29.765411 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:29.797606 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:29.799664 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:29.826550 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:29.864007 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:21:29.895869 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:29.924137 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:21:29.925311 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:29.925594 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:29.933887 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:29.941018 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:29.999881 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:30.012842 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:30.023012 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:30.069197 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:30.072680 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 08:21:30.172335 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:30.313112 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:30.323200 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:21:30.330687 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:30.333333 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:30.347394 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:30.365259 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:30.477484 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:30.480325 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:30.481010 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:21:30.552834 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:30.561369 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:30.575713 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240501 08:21:30.590495 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:30.596136 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:30.647380 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:30.673389 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:21:30.718014 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:30.862565 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:30.960316 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:30.964092 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:21:30.987728 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:30.994652 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:31.003707 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:31.008647 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:31.163826 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:31.166281 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:21:31.201599 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:31.203678 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:31.204352 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:31.208792 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:21:31.215859 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:31.229974 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:31.241873 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:31.450886 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:31.573753 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:31.624519 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:31.643180 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:31.643536 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:31.644362 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:31.709425 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:31.710352 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:21:31.830940 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:31.846689 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 08:21:31.855065 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:31.865061 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:31.866422 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:31.867769 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:31.869066 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:31.936743 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:31.937808 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 08:21:31.948736 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:31.956370 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:32.004062 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 08:21:32.040994 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:32.097752 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:32.190909 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:32.229923 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:21:32.257493 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:32.270069 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:32.273233 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:32.291219 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:32.309983 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:32.317852 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:32.318935 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:32.329511 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:32.373857 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:32.388904 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:32.390959 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:32.397044 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:32.447449 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:21:32.469763 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:32.478133 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:32.484972 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:32.486567 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:32.537125 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:21:32.550978 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:32.577679 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:32.706903 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:32.736019 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:32.784582 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240501 08:21:32.903627 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:32.913955 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:32.921054 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:32.947314 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:21:32.949012 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:32.970135 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:32.991667 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:33.044749 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:21:33.138455 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:33.147912 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:33.156245 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:33.163923 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:33.285614 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:33.336459 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:33.374511 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 08:21:33.375332 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:33.388684 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 08:21:33.393671 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:33.432075 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:33.544781 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:33.555291 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:33.560714 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:33.561712 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:33.571794 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:33.600311 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:33.657783 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:33.722348 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:33.757138 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:21:33.773053 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:33.781564 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:33.789604 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:33.803332 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:33.995952 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:34.014747 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:34.063989 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240501 08:21:34.146123 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:34.153443 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240501 08:21:34.185258 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:34.194183 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:21:34.201027 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:34.205567 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:34.228552 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:34.309389 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:34.366953 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:34.368209 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:34.398123 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240501 08:21:34.398897 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:34.417359 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:34.421142 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:34.431684 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:34.467793 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:34.468401 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:34.469971 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:34.541805 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:34.560989 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:34.607549 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:34.615869 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:34.650058 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:34.717983 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:34.759871 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:34.764506 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240501 08:21:34.837150 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:34.844520 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:34.848263 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:34.850425 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:34.861038 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:34.902911 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:34.957193 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:34.963915 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:21:35.052192 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:35.063338 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:21:35.069918 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:35.075321 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:35.078452 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:35.083021 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:21:35.137377 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:35.147689 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:35.202491 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:35.234786 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:35.241341 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:35.275655 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240501 08:21:35.464391 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:35.474524 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:35.483289 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:35.497325 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:35.534237 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:35.636682 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:35.651160 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:21:35.677415 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:35.677815 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:21:35.682509 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:35.692554 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:35.717867 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:35.759864 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:35.823730 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:35.848083 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:35.848241 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:35.920342 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:35.943737 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:36.021656 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:36.094589 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:21:36.117569 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:36.125227 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:36.137161 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:36.152917 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:36.196523 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:36.233037 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 08:21:36.248692 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:36.333891 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:36.344525 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:36.355326 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:36.361426 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:36.362948 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:36.425294 9536 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:21:36.425521 9536 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:21:36.426198 9536 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:21:36.427959 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:36.428145 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:21:36.428277 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:21:36.428534 9536 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:21:36.494436 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:36.758980 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:36.759539 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240501 08:21:36.765610 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:36.780090 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:36.781077 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240501 08:21:36.798404 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:36.810750 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:36.832419 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:36.861373 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:36.913807 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 08:21:36.970474 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:36.972894 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:36.983104 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:36.989138 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:36.993196 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:37.009727 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:37.040262 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:37.053637 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:37.084117 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:37.108776 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:37.166165 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:37.190109 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:37.252179 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:37.258858 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:37.348258 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:37.350700 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:21:37.391052 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:37.425170 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:37.439975 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:37.441078 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:37.447871 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:21:37.469199 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:37.495282 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:37.506172 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:37.522347 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:21:37.599742 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:37.605751 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:37.607416 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:37.621510 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:37.628118 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:37.666972 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:37.715121 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:37.737195 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:37.778555 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:37.797052 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 08:21:37.867138 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:37.945487 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:21:38.021085 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:38.021682 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:38.035540 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:38.036574 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:38.043036 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:38.053776 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:21:38.219868 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:38.230391 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:38.233754 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:38.251648 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:38.258373 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:38.271721 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:38.317414 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:38.355283 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:38.393352 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 08:21:38.394162 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:38.413622 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:38.460954 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:38.483444 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:38.595464 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240501 08:21:38.653666 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:38.669143 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:38.671325 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:38.673434 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:38.674327 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:38.742674 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:38.813243 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:38.891492 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:38.903723 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:38.930054 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:38.944504 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:39.018126 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:39.079947 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 08:21:39.119681 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:39.176524 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:39.184192 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 08:21:39.187366 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 08:21:39.219208 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:21:39.323864 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:39.330204 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:39.335439 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:39.351495 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:39.382707 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:39.394874 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 08:21:39.409977 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:21:39.461606 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:39.486908 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:39.506049 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:21:39.568698 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:39.578088 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:39.592286 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:39.595817 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:39.599671 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:39.654330 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:39.674369 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:39.693373 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:39.722892 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:39.803407 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:39.837404 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:39.843053 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:39.880761 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:39.923976 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:39.934824 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:39.943265 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:21:40.000049 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:40.017802 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:40.018925 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:40.026249 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:40.036307 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:40.061877 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:40.065986 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:40.076632 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:40.121030 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:40.185326 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240501 08:21:40.206678 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:40.212044 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:40.226482 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:40.243968 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:40.261494 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:40.364876 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:40.369972 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:40.479787 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:40.493104 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:40.532642 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:40.534498 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:21:40.623471 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:40.628731 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:40.634030 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:21:40.639935 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:40.649415 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:40.734212 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:21:40.735121 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:40.820377 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:40.834318 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:40.839560 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:21:40.845870 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:40.849293 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:40.861150 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:40.862882 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:41.024406 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:41.062042 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:41.121111 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 08:21:41.184140 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:41.247027 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:41.249379 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:41.258541 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:41.262462 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:41.367825 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 08:21:41.375850 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:21:41.453300 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:41.470597 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:41.472530 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:41.479321 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:41.487720 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:41.535799 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:41.585717 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:41.695739 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 08:21:41.751659 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:41.872126 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:41.873603 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:41.886206 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:41.890033 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:41.928378 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:41.965088 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:41.981665 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:41.996219 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:41.999497 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:42.009543 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:42.024554 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:42.040616 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 08:21:42.061231 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:42.079492 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:42.087188 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:42.088739 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:42.115662 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:42.133482 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:42.158212 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:42.158170 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:21:42.313485 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:42.410737 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:42.411343 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:42.480058 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:42.489475 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:42.496210 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:42.506196 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:42.518139 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:42.520386 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:42.522346 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:42.550333 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:42.550568 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:21:42.566376 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:42.648032 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:42.722751 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:21:42.734831 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:42.750612 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:42.752810 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:21:42.757000 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:42.758419 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:42.776098 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:42.840966 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 08:21:42.901242 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:42.958894 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:43.068979 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:43.089198 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:43.143242 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:43.166373 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:43.167289 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:43.178169 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:43.197806 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:43.351905 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:43.361518 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:43.370699 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:43.374857 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:43.379922 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 08:21:43.400527 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:43.404014 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 08:21:43.405071 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:43.491528 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:43.509397 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:43.532374 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 08:21:43.568027 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:43.568377 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:43.626279 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:43.803014 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:43.809407 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:43.814932 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:43.818578 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:43.822402 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:43.891907 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:44.000507 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:44.001060 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:44.021808 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:44.026711 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:44.039472 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:44.083573 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 08:21:44.113762 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:44.218084 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:44.257143 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:21:44.257910 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 08:21:44.299890 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:21:44.314792 11080 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:21:44.315007 11080 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:21:44.315646 11080 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:21:44.317128 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:44.317341 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:44.317449 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:21:44.317679 11080 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:21:44.416389 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:44.419044 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:44.429394 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:44.436980 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:44.477516 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:44.505137 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:44.607103 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:44.608314 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:21:44.612207 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:44.626060 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:44.632885 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:44.634341 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:44.646499 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:44.647909 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:44.682648 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:44.705026 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:44.725638 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:44.732898 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:44.734440 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:44.801124 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:44.831482 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 08:21:44.909914 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:45.037490 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:45.037875 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:45.042666 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:45.064472 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:45.068076 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 08:21:45.070400 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:45.073221 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:45.124482 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:45.152159 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:45.153963 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:45.173885 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:45.245127 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:45.250123 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:45.251886 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:45.262022 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:45.268774 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:45.330242 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:45.374068 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:45.439972 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:45.441723 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:21:45.676265 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:45.676872 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:45.681421 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:45.693651 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:45.695122 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 08:21:45.695624 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:45.699798 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:45.714408 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:45.754290 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:21:45.773900 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:45.837078 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:45.866134 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:45.887867 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:45.891831 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:45.900521 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:45.920558 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:45.945839 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:45.950228 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:46.005867 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:46.064409 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:46.171453 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:46.271590 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240501 08:21:46.279064 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:46.290709 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:46.297506 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:46.307101 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:46.439087 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:46.465806 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:46.481909 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:46.482010 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:46.493232 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:46.500978 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:46.641369 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:46.716919 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:46.796036 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:21:46.870465 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:46.881637 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240501 08:21:46.884595 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:46.885190 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:46.890389 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:46.984694 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:46.997164 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:47.008930 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:47.082803 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:47.083287 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:21:47.092491 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:47.106508 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:47.106973 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:47.148638 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:47.158754 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:47.168833 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:47.172808 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:47.263633 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:21:47.308493 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:47.325450 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:47.384315 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:47.398963 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:47.417548 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:47.496124 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:21:47.503311 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:47.515012 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:47.515442 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:47.528290 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:47.554900 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:47.609184 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:47.630316 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:47.633630 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 08:21:47.714890 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:47.721195 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:47.731107 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:47.764532 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:47.777179 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:47.777695 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:47.778461 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:47.807401 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 08:21:47.936820 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:48.017287 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:48.051762 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:48.061455 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 08:21:48.079985 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:48.105904 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:48.160039 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:48.167762 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:48.179759 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:48.181991 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:48.190364 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:48.194802 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:48.215857 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:48.220732 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:48.352139 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:21:48.367655 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:48.368643 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:48.378443 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:48.391515 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:48.426554 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240501 08:21:48.461408 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:48.498677 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:48.533710 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:48.644151 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:48.763137 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:48.805100 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:48.808032 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:48.814580 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:48.822484 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:48.929958 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240501 08:21:48.994827 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:49.019868 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:49.033886 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:49.037288 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:49.046988 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:49.065487 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:49.094944 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 08:21:49.195379 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:49.280417 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:49.309615 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:21:49.310468 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:49.334317 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:49.383570 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:49.383858 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240501 08:21:49.439334 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:49.445189 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:49.447093 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:49.459301 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:49.565227 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:49.578760 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:49.634208 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:21:49.637053 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:49.644019 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:49.655233 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:21:49.658990 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:49.663034 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:49.720517 6573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.107.4:43623, user_credentials={real_user=slave}} blocked reactor thread for 49653us
W20240501 08:21:49.726774 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:49.728349 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:49.756523 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:21:49.818766 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:49.857796 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:49.918857 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:49.965157 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:49.966382 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:50.006224 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 08:21:50.013860 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:50.041405 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:21:50.116808 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:50.121385 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:21:50.139743 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:50.140157 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:50.146411 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:50.210031 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:50.247108 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:50.266219 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:50.286324 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:50.321821 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:50.329931 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:50.330964 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:50.341836 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:50.342861 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:50.467375 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:50.476207 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 08:21:50.488669 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 08:21:50.615942 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:50.623191 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:50.679610 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:50.735282 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:21:50.757975 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:50.762562 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:21:50.771611 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:50.775272 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:50.778944 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:50.844694 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:21:50.845484 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:50.854809 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:50.865727 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:50.884688 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:50.959774 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:50.964874 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:50.965808 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:50.972780 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:51.017853 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:51.159240 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 08:21:51.229319 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:51.273443 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:51.368312 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:51.377226 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:51.384315 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:51.386598 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:51.388782 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:51.599946 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:51.600382 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:51.610508 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:51.614194 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:51.630813 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:51.735849 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:51.772468 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:51.811997 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:51.832911 10808 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:21:51.833097 10808 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:21:51.833792 10808 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:21:51.835381 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:51.835610 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:21:51.835721 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:21:51.835937 10808 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:21:51.836115 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 08:21:51.865805 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:51.894519 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:51.981456 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:21:52.030889 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:52.040102 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:52.042532 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:52.043926 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:52.060170 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:52.070861 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:52.139755 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:52.140035 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:52.191699 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:21:52.197055 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:52.204639 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:52.237066 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:52.237090 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:52.248171 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:52.252879 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:52.275574 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:52.333945 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:52.344198 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:52.402827 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:52.434005 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 08:21:52.438624 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:52.483162 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:52.605377 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:52.646844 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:52.664499 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 08:21:52.667285 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:21:52.669337 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:52.676559 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:52.683058 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:52.710825 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:52.716141 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:52.718592 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:52.771600 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:52.776896 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:52.811230 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:52.885924 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:52.886956 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:52.891254 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:52.894438 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:53.100927 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 08:21:53.148609 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:53.202629 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:53.205112 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:53.302248 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:21:53.305397 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:53.310144 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:53.316963 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:53.318861 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:53.322737 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:53.331517 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:53.342448 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:53.346787 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:53.359473 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:53.424297 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:53.485706 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240501 08:21:53.516052 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:53.520241 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:53.534210 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240501 08:21:53.536320 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:53.546533 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:53.548092 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:53.726367 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:53.799767 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:53.935217 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:53.949440 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:53.953308 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:53.954628 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:53.967060 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:54.116573 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 111 similar messages]
I20240501 08:21:54.163669 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:54.180827 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:54.182591 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:54.198833 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:54.200546 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:54.253610 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:54.328126 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:21:54.387506 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 08:21:54.436115 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:54.445739 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:54.447564 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:54.488651 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:54.520702 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:54.523883 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:54.525738 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:21:54.575165 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:54.595902 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:21:54.609493 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:54.617673 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:54.621882 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:54.626880 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:54.636672 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 08:21:54.690901 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:54.757596 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:54.765645 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:54.797763 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:21:54.812961 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:54.818681 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:54.829535 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:54.836897 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:21:54.841153 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:54.894503 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:54.932521 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:54.952198 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:54.987649 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:55.052907 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:55.057164 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:55.179175 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:21:55.181226 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:55.196377 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:55.248492 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 08:21:55.248580 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:55.248759 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:55.261967 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:55.262040 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:55.273708 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:21:55.291836 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:55.387888 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:55.450197 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:55.454573 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:55.454784 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:55.477926 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:55.483566 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:55.574899 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:21:55.623404 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:55.678570 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:55.679378 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:55.778993 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:55.826732 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 08:21:55.840858 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:21:55.859210 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:55.865646 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:55.869210 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:55.885489 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:55.895187 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:21:55.927332 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:55.942050 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:21:55.993330 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:56.087563 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:56.091219 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:56.107712 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:56.111802 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:56.114472 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 08:21:56.220700 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:56.241907 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:56.468917 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:56.499835 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:56.507673 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:56.510075 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:56.523324 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:56.641609 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:56.694756 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:56.720793 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:21:56.740808 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:56.744395 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:56.751577 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:56.767899 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:56.782905 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 08:21:56.857539 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:56.919853 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:21:56.955197 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:57.037266 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:57.044747 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:57.086141 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:21:57.125569 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:57.136173 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:21:57.156313 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:57.165134 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:57.169585 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240501 08:21:57.174852 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:57.178544 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:57.180469 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:57.181779 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:57.210603 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:57.360013 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:57.387056 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:57.390475 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:57.393503 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:57.491590 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:57.504343 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:21:57.506960 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:57.522969 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:57.583642 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:21:57.613612 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:57.623195 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:21:57.682137 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:57.731395 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:57.745980 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:57.772915 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:21:57.804183 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:57.813974 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:57.831215 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240501 08:21:57.837893 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:57.842530 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:57.855670 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:21:57.899850 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:21:57.997313 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:21:58.026199 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:58.048712 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:58.056193 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:58.062809 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:58.089890 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:58.179271 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:21:58.182909 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:21:58.184898 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:58.231684 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:58.276751 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:21:58.346637 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:58.439594 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:58.444232 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:58.447616 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:21:58.451282 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:58.458413 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:58.460228 13253 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:21:58.460394 13253 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:21:58.461036 13253 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:21:58.466158 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:58.466650 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:58.466739 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:21:58.470410 13253 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:21:58.471175 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:58.494391 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240501 08:21:58.520711 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 08:21:58.615837 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240501 08:21:58.692119 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:58.692592 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:21:58.707674 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:21:58.722023 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:58.724936 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:58.797348 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:21:58.842927 6486 maintenance_manager.cc:419] P d978dc7272e947d9938cb39d0d6c1eb8: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033396
I20240501 08:21:58.848552 6485 maintenance_manager.cc:643] P d978dc7272e947d9938cb39d0d6c1eb8: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.003s sys 0.000s Metrics: {"bytes_written":2533,"delete_count":0,"lbm_write_time_us":151,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":211,"threads_started":1,"update_count":53,"wal-append.queue_time_us":317}
I20240501 08:21:58.849110 6486 maintenance_manager.cc:419] P d978dc7272e947d9938cb39d0d6c1eb8: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.174690
W20240501 08:21:58.854178 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:58.862293 6485 maintenance_manager.cc:643] P d978dc7272e947d9938cb39d0d6c1eb8: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.012s sys 0.001s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15168,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":576,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":134,"lbm_read_time_us":265,"lbm_reads_lt_1ms":17,"lbm_write_time_us":293,"lbm_writes_lt_1ms":9,"reinsert_count":0,"spinlock_wait_cycles":896,"thread_start_us":405,"threads_started":2,"update_count":53}
W20240501 08:21:59.086455 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:21:59.125631 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:59.134781 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:59.153065 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:21:59.153998 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:21:59.182425 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:59.248742 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:21:59.257380 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 08:21:59.326336 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:21:59.333982 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:21:59.350190 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:59.359409 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:59.366345 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:59.389024 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:59.410387 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:21:59.460352 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:21:59.465889 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 08:21:59.521059 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:21:59.521847 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:59.587293 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 08:21:59.686661 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:21:59.735010 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:21:59.739025 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:21:59.768049 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:59.774413 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:59.778645 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:59.779819 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:21:59.816535 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:21:59.819221 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:21:59.878703 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:21:59.889050 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:21:59.963900 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:21:59.973253 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:21:59.993614 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:00.006964 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:00.008924 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:22:00.023653 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:00.111075 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:00.112880 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:22:00.128710 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:00.252512 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:00.252512 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:22:00.270989 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:00.296036 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:00.303069 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:00.317221 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:22:00.380774 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:00.385627 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:00.387148 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:00.390344 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:00.392091 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:00.437984 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:00.449187 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:00.454041 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240501 08:22:00.584278 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:00.587105 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:00.595921 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:00.601394 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:22:00.610175 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:00.614199 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:00.720320 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:22:00.780341 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:00.993091 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:22:00.994606 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:00.997403 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:00.998247 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:01.006922 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:01.023299 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:01.023984 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:01.042068 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:22:01.107489 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 08:22:01.124403 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:01.189745 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:01.203774 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:22:01.206918 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:01.221192 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:01.234493 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:01.255869 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:01.419010 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 08:22:01.476291 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:22:01.523347 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:01.602136 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:01.602135 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:01.618016 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:01.632164 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:01.659544 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:22:01.707624 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:01.753001 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:01.824240 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:01.833743 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:01.834038 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:22:01.844058 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:01.855142 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:02.058672 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:02.117622 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:02.132464 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 08:22:02.133332 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 08:22:02.186537 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:22:02.225677 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:02.252887 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:22:02.258957 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:02.259191 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:02.265751 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:02.275694 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:02.284147 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:02.300096 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:02.319264 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:22:02.323678 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:22:02.353439 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:02.405310 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:22:02.431826 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:02.446156 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:02.446934 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:02.447610 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:02.452919 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:22:02.464776 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:02.613691 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:02.658113 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:02.680369 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:02.751235 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:02.766568 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:02.781270 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:02.785528 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:02.801553 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:22:02.821712 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:02.838706 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:02.840718 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:02.851305 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:02.861773 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:02.906013 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:02.913354 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:02.954636 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:02.983289 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 08:22:02.992343 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:22:03.050352 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:03.061136 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:03.067518 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:03.075719 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:03.195494 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:03.215567 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:03.229133 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:03.428354 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:22:03.452220 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:03.455265 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:03.460012 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:22:03.472085 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:03.488859 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:03.510452 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240501 08:22:03.655028 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:03.681250 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:22:03.686631 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:03.688071 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:03.690622 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:03.690778 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:03.719436 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:03.744262 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 08:22:03.780555 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:04.034471 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:22:04.104281 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:04.114295 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:04.134527 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:04.141604 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:04.188264 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:22:04.247602 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:04.265000 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:04.265509 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 08:22:04.266557 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 08:22:04.285725 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:04.287024 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:22:04.294412 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:04.306114 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:04.313784 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:04.328469 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:04.461727 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:22:04.462661 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 08:22:04.493412 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:22:04.638708 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:22:04.689963 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:22:04.696091 6484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240501 08:22:04.698513 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:22:04.737609 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240501 08:22:04.753652 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:04.757100 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:04.759148 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:04.765976 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:04.766515 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:04.803524 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:04.832054 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:04.848726 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:22:04.865746 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:04.879801 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:04.898701 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:04.902890 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:22:04.962168 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:04.971971 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:22:04.974426 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:04.977391 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:04.978492 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:05.097314 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:22:05.117619 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:05.161576 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:05.176002 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:05.213363 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:05.225978 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:05.231875 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:05.328289 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:22:05.354630 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:05.357019 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:05.366438 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:05.374166 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:05.376320 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:05.399129 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:05.400491 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:05.465166 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:05.476858 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:05.515264 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240501 08:22:05.589025 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:05.598804 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:05.602794 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:05.604220 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:05.702471 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:05.789017 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:05.811012 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:05.889796 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:05.908345 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:22:05.978785 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:06.020972 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:06.038652 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:06.046738 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:06.048722 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:06.072341 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:22:06.152745 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 08:22:06.169559 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:22:06.228055 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:06.232663 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:06.243207 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:06.246716 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d7903106d7e74e1ea102b1abdbfce63e: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:22:06.252573 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:06.271054 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:06.312006 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:06.590075 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:06.647544 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:06.658561 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:06.662990 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:06.666373 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 08:22:06.669520 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:22:06.683109 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:06.787020 14051 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:22:06.787232 14051 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:22:06.787880 14051 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:22:06.789495 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:06.789705 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:06.789808 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:22:06.790094 14051 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:22:06.857501 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:06.865047 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:06.867076 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:06.876078 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:06.885092 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:06.891932 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:06.907172 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 08:22:06.911435 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:22:06.945072 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:06.945749 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:07.034718 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:07.040123 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 08:22:07.119066 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:07.204988 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:07.237596 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:22:07.238334 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:07.274165 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:22:07.294795 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:07.304077 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:07.305651 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:07.316946 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:07.321485 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:07.351882 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:07.354919 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:07.462409 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:07.493395 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:07.504281 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:07.510908 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:07.512444 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:07.572144 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:07.596010 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:07.625320 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:07.650517 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:07.657253 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:07.666791 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:07.695703 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:07.733752 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 193de3c11ae54408a64ae3007bf88920: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:07.735347 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:07.779116 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:07.841547 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:07.888727 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240501 08:22:07.912784 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:07.923736 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:07.925035 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:07.925081 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:07.944638 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:07.989208 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:08.032450 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:08.057641 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:08.166208 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:08.179920 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:08.182781 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:08.183143 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:08.232656 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:08.233574 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:08.276971 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:08.300220 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:08.334604 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:08.468683 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:08.511209 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 08:22:08.546279 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 08:22:08.550490 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:08.552105 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:08.588903 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:08.601498 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:08.620774 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:08.621203 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:08.702739 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:08.794425 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:08.802004 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:08.803354 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:08.817425 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:08.946429 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:08.993021 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:09.072796 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:09.130712 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:09.220588 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:09.228710 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:09.243801 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:09.244673 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:09.298081 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 08:22:09.310745 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 08:22:09.332829 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:22:09.377097 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:09.443352 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:09.444574 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:09.448357 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:09.459757 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:09.464231 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:22:09.465377 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 08:22:09.475193 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:09.480635 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:09.536567 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:09.540977 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:09.542929 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:09.710295 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:09.715833 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:09.757241 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 08:22:09.779844 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:09.786239 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:09.798410 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:09.836145 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:22:09.846940 14401 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:22:09.847139 14401 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:22:09.847884 14401 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:22:09.850993 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:09.853644 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:09.856269 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:22:09.856431 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:22:09.857797 14401 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:22:09.861085 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:09.863405 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:09.867218 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:09.870332 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:09.950390 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:10.052351 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:10.053465 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:10.069218 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:10.069775 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:10.084767 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:10.240950 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:10.263258 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:10.263981 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:10.268790 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 08:22:10.297396 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:10.324435 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:10.327286 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:10.333102 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:10.336849 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:10.371465 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:10.401510 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:22:10.470784 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:10.472291 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:10.488282 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:10.491019 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:10.519428 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:10.563620 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:10.607005 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:10.634455 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:22:10.684531 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:10.709671 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:10.727646 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:10.732586 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:10.770468 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:10.827330 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:10.857321 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:10.932041 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:11.031709 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:11.094151 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:11.134248 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240501 08:22:11.159494 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:11.161787 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:11.175791 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:11.190025 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:11.230930 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:22:11.256429 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:11.373157 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:11.387820 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:11.389645 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:11.389678 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:11.460325 14051 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:22:11.460515 14051 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:22:11.461174 14051 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:22:11.462648 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:11.462843 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:11.462956 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:22:11.463239 14051 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:22:11.503592 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:11.551353 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:11.569353 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:11.760145 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:11.784986 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:11.801441 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240501 08:22:11.804723 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:11.805747 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:11.824754 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:11.840098 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:11.876593 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:11.882774 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:11.940027 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:11.982793 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:11.990628 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 08:22:12.013919 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:22:12.033845 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:12.047261 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:12.058058 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:12.060633 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:12.065518 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:12.229457 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:12.253772 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 08:22:12.297379 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:12.321460 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:12.376971 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:12.386363 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:22:12.431715 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:12.439714 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:12.453238 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:12.462764 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:12.478999 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:12.484927 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:12.485724 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:22:12.487855 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:12.585958 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240501 08:22:12.652143 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:12.662624 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:22:12.677801 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:12.684098 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:12.711294 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:12.792834 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:12.796373 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:12.851239 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:12.928225 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:12.929454 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:12.931777 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:12.948189 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:12.956326 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:12.960817 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:13.007113 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:13.010876 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:13.084278 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:13.094532 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:13.097553 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:13.098980 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:13.125032 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:13.179414 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:13.198356 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:22:13.271348 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:13.286715 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:13.289695 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:22:13.298816 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:13.307600 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:13.404987 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 08:22:13.454629 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:13.487452 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:13.496754 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:13.593021 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:13.615957 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240501 08:22:13.754910 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:13.767110 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:13.770931 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:13.777573 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:13.940140 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:13.994196 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:14.000217 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:14.007370 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:14.019022 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:22:14.023831 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:14.049611 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:14.057016 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:14.188593 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 08:22:14.257470 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:14.361364 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:14.382743 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:14.386977 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 111 similar messages]
I20240501 08:22:14.412670 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:14.416807 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:14.419634 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:14.424984 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:14.430929 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:14.432389 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:14.443459 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:14.460983 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:14.473165 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240501 08:22:14.480587 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:14.624260 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:14.632526 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:22:14.637291 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:14.639555 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:14.652339 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:14.660742 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:14.729645 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:14.774958 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:14.801424 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 08:22:14.845249 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 08:22:14.880580 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:14.949831 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:14.978976 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:14.981472 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:22:14.991765 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:15.056494 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 08:22:15.098748 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:22:15.100656 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:15.112778 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:15.125247 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:15.135417 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:15.210820 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:15.229451 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:15.284860 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:15.295001 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:22:15.309990 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:15.315739 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:15.319638 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:15.326526 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:15.367377 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:15.371659 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:15.396667 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:15.397491 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:15.416429 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:15.464224 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:15.485355 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:15.719741 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:15.728662 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:22:15.738644 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:15.749002 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:15.752422 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:15.776355 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:15.800721 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:15.816617 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:15.914023 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 08:22:15.949115 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:22:15.952045 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:15.960848 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:15.971771 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:15.976410 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:15.995352 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:16.077137 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:16.229255 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:16.272015 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:16.293869 14887 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:22:16.294118 14887 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:22:16.294912 14887 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:22:16.295492 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240501 08:22:16.296739 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:16.296983 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:16.297093 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:22:16.297353 14887 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:22:16.298767 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:22:16.400326 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:16.407750 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:16.418475 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:16.423333 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:16.487746 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:16.553606 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:16.577608 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:16.600340 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:16.602947 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:16.616138 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:16.618492 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:16.621315 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:16.775640 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 08:22:16.781682 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:16.827749 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:16.836858 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:16.875180 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:16.949842 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:22:17.017925 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:17.019358 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:17.032594 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:17.044889 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:22:17.046324 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:17.120486 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:17.139837 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:17.149379 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 08:22:17.154223 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:22:17.225621 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:17.226621 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:17.236095 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:17.248003 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:17.315387 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 08:22:17.375747 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:17.390949 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:17.474607 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:22:17.477036 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:17.490787 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 08:22:17.522229 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:17.581941 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:17.583146 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:17.625672 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:17.625934 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:22:17.628939 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:17.644764 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:17.646510 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:17.658360 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:17.663156 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:17.691602 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:17.737910 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:17.743336 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:17.787899 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:22:17.850358 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:17.855870 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:17.865446 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:17.866428 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:17.993804 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:18.003645 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:18.071712 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:18.186409 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:18.207751 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:18.241015 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:18.255493 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:18.267907 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:18.270059 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:18.280885 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:18.411029 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 08:22:18.452908 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:22:18.476176 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:18.478639 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:18.485672 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:18.487164 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:18.569948 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:18.573813 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:18.656311 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 08:22:18.676304 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:18.691895 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:18.781493 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:18.899904 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:18.906116 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:18.908524 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:18.909685 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:18.950729 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 08:22:18.968897 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:19.069799 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:19.100790 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:19.103755 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:19.125977 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:19.126076 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:22:19.127347 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:19.156725 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:19.197062 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:19.203168 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:19.209178 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:19.221724 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:19.397543 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 08:22:19.408632 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:19.458101 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 08:22:19.465411 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:19.492599 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:19.493654 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:22:19.525593 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:22:19.559823 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:19.572039 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:19.577045 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:19.579167 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:19.580590 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:19.643536 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:19.647995 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:19.717645 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:22:19.733150 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:22:19.766283 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:19.771692 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:19.776546 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:19.778110 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:19.800964 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:19.911831 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 08:22:19.913141 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 08:22:19.964931 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:20.012167 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:20.049405 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:20.055720 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:20.056347 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:20.063935 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:20.168967 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:20.178982 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:22:20.185128 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:20.191466 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:20.212880 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:20.213948 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:20.236418 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:20.246891 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:22:20.425729 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:20.438427 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:20.439857 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:20.442880 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:20.481546 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:20.495734 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:20.620666 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:20.641988 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:20.677896 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:20.754774 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:20.797811 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:22:20.826200 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:20.836753 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:20.840394 15600 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:22:20.840571 15600 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:22:20.841302 15600 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:22:20.847066 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:20.847275 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:20.847345 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:22:20.847601 15600 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:22:20.852008 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:20.860564 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:20.933706 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:21.014919 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240501 08:22:21.036979 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:21.048530 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:21.056665 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:21.058908 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:21.104948 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:21.219834 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:21.242463 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:21.270128 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:21.319715 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:22:21.369822 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:21.370575 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 08:22:21.433393 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:22:21.460376 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:21.475391 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:21.476511 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:21.480849 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:21.497148 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:21.547994 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:21.598181 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:21.691052 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:21.693385 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:21.701958 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:21.708182 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:21.714998 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:21.759300 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:21.861841 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 08:22:21.879566 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:21.885720 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:21.922552 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:21.981135 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:22.025648 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:22.051638 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:22.083006 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:22:22.137820 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:22.154595 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:22.170524 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:22.173552 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:22.175736 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:22.185657 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:22:22.187841 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:22.265625 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:22:22.281375 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:22:22.355188 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:22.370281 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:22.370515 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:22.387799 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:22.428416 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:22.443526 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:22.473464 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:22.473531 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:22.487632 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 08:22:22.488862 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:22.533357 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 08:22:22.575958 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:22.674335 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:22.712939 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:22.726330 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:22.734241 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:22.799017 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:22.819048 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:22.823808 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:22.829833 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:23.008244 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:23.021317 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:23.025110 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:23.034926 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:23.043321 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:23.043365 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:23.089205 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:23.317651 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:23.327483 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:23.425364 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:22:23.472146 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:23.477790 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:23.487998 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:23.488927 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:23.497308 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:23.537678 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 08:22:23.545542 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:23.589223 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:23.626367 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:23.686805 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:23.689971 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240501 08:22:23.692286 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:23.725426 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:23.733242 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:23.759398 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:23.800293 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:23.838147 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 08:22:23.946045 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:23.952577 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:23.997121 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:24.028170 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:24.066540 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:22:24.136410 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:24.154732 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:24.161811 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:24.170831 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:24.218385 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:24.269418 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:24.319752 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 08:22:24.339327 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:22:24.373389 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:24.390635 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:24.400132 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 110 similar messages]
I20240501 08:22:24.411190 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:24.417119 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:24.423427 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:24.467864 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:24.512396 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:22:24.541448 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:24.558820 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:24.612624 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:24.614235 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:24.697955 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:24.796638 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:24.824569 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:24.830857 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:24.856041 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:24.857007 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:22:24.883179 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:24.911116 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 08:22:24.940174 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 08:22:24.959499 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:24.977073 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:24.987643 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:24.993947 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:25.026834 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:22:25.048430 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:25.053660 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:25.055732 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:25.072240 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:25.076779 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:25.143611 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:25.158871 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:25.168926 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:25.218312 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:25.223374 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:25.433835 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:22:25.451084 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:25.456128 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:25.459581 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:25.466908 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:25.482035 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:25.541446 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:25.544842 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:25.680874 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:25.682053 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:25.692090 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:25.724596 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:25.744048 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:25.923525 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:26.025040 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:26.045565 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 08:22:26.046351 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:26.070820 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:26.075280 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:26.104467 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:26.139962 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:22:26.184230 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:26.192831 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:26.203334 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:26.215736 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:26.221683 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:26.222622 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:26.263355 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 08:22:26.322837 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:22:26.346360 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:22:26.402242 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:26.403498 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:26.411031 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:26.420804 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:26.449424 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:26.608016 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:26.613135 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:26.683100 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 08:22:26.691967 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:26.777315 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:22:26.809089 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:26.815821 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:26.815832 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:22:26.828071 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:26.833999 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:26.915148 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:27.002665 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:22:27.020057 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:27.022089 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:27.034718 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:27.034940 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:22:27.044031 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:27.055709 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:27.074725 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:27.105018 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:27.135524 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:22:27.135987 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:27.191751 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:27.367702 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:27.388720 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 08:22:27.401217 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:27.459022 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:27.459153 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:27.470619 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:27.476097 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:27.480314 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:27.491250 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:27.506067 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 08:22:27.527468 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:27.543607 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:27.554105 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:27.576759 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:27.648928 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:27.673485 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:27.687386 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:27.709923 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:22:27.717265 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:27.732251 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:27.877488 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:27.914296 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:28.022459 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:28.059792 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:28.075760 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:28.107950 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:28.111984 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:22:28.124537 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:28.126068 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:28.129105 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:28.132202 16669 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:22:28.132413 16669 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:22:28.133213 16669 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:22:28.147354 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:22:28.148938 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:28.151059 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:22:28.151691 16669 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:22:28.276129 16627 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:22:28.276335 16627 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:22:28.277107 16627 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:22:28.282387 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:28.282634 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:28.282768 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:22:28.283046 16627 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:22:28.322507 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:28.322752 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:28.325817 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:28.338068 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:28.515084 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 08:22:28.521430 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:28.538875 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:28.541193 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:28.588645 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:28.615702 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:28.664137 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 08:22:28.700587 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 08:22:28.743746 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:28.772287 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:28.809170 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:22:28.810242 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:28.824316 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:28.829156 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:28.835397 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:28.843817 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:28.846815 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:28.952383 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:29.013132 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:29.017782 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:22:29.022953 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:29.028685 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:29.032039 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:29.045938 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:29.081053 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:22:29.127099 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:29.171489 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:29.227259 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:29.420621 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240501 08:22:29.421624 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:22:29.432898 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:29.436091 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:29.447961 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:29.459376 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:29.520804 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:22:29.562331 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:29.565647 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 08:22:29.574801 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:29.606462 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:29.634887 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:22:29.641883 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:29.655745 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:29.655820 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:29.697328 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:22:29.701331 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:29.711422 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:29.804522 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:29.815688 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 08:22:29.851181 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:29.882216 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:29.945677 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:29.993038 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:30.011292 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 08:22:30.071889 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:22:30.093004 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:30.093472 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:30.110705 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:30.129544 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:30.167508 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:30.169663 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:30.269001 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:22:30.322710 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:30.328032 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:30.332847 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:30.336642 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:30.342126 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:22:30.345384 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:30.445044 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:30.450109 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:30.515491 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:30.564556 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:30.647979 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:30.705771 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:30.744822 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:30.767333 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240501 08:22:30.779819 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:30.783581 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:30.794565 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:30.895206 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:30.969758 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:30.978680 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:30.979635 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:30.996254 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:31.022246 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:31.055568 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:31.057806 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:31.150018 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 08:22:31.168033 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:31.208899 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:31.220618 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:31.339751 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:22:31.348520 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:31.349310 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:31.358575 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:22:31.389025 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:31.402068 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:31.403807 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:31.406787 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:31.415524 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:31.436471 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:31.582222 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:31.612952 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:31.633098 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:31.638096 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:31.642485 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:31.642817 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:31.676244 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:31.714339 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:31.839888 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:31.850693 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:31.921305 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:32.000818 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:22:32.060091 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:32.069481 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:32.069851 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:22:32.070715 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:32.078625 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:32.096534 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240501 08:22:32.100364 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:32.135252 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:32.206890 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:22:32.216050 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:32.226748 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240501 08:22:32.263782 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:32.267067 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:32.276628 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:32.294939 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:32.347874 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:32.489915 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:32.494233 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:32.508002 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:32.513736 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 08:22:32.637462 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:32.661052 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:32.667929 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:32.677038 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:22:32.695266 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:32.699961 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:32.738835 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:32.792325 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:32.841845 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:32.852576 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:32.886152 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:32.899799 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:32.907477 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:32.925360 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:32.926292 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:32.968628 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:33.032744 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:33.072841 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:33.072965 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:33.199435 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 08:22:33.225798 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:33.249788 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:22:33.303103 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:33.317827 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:33.321626 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:33.333320 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:33.440827 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:33.449165 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:33.492053 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240501 08:22:33.494596 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:33.502943 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:33.515818 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:33.521677 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:33.530908 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:33.544173 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:33.630183 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:33.667764 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:33.717943 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 08:22:33.812728 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:22:33.918226 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:33.937707 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:33.946163 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:33.947345 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:22:33.950510 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:33.963276 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:34.010073 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:34.047479 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:34.075064 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:34.086318 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:34.112185 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:22:34.169760 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:34.173429 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:34.191911 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:34.196640 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:34.216116 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:34.216192 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:34.267160 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:34.288990 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:34.354771 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:34.428200 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240501 08:22:34.465493 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:34.497766 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 08:22:34.515633 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:34.547744 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:22:34.610409 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:34.623111 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:34.626463 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:22:34.628505 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:34.634238 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:34.645443 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240501 08:22:34.658566 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:34.822793 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:22:34.830142 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:34.851673 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:34.856565 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:34.859637 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:34.879719 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:34.924998 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:22:34.925426 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:34.967644 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 08:22:34.996691 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:35.017386 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:35.077497 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:35.187407 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:22:35.250862 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:35.271478 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:35.276995 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:35.279217 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:35.303426 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:35.335960 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:35.370216 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:35.379130 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:22:35.396399 17153 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:22:35.396606 17153 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:22:35.397388 17153 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:22:35.399152 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:35.399359 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:35.399452 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:22:35.399719 17153 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:22:35.403057 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:22:35.463089 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:35.478190 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:35.478802 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:35.485601 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:35.492337 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 08:22:35.530134 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:35.571856 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:35.621577 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:35.686928 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:35.824321 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 08:22:35.848320 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:22:35.879742 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:35.890713 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:22:35.904474 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:35.918859 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:35.927167 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:35.932442 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:35.991952 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:36.011761 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:36.104184 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:22:36.131376 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:36.132125 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:36.165190 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:36.178917 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:22:36.180300 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:36.189838 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:36.360939 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:22:36.361711 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:36.363875 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:36.409020 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:36.439962 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:36.465263 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:36.516388 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:36.563750 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:36.574885 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:36.590799 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:36.591496 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:36.606570 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:36.638823 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:22:36.772277 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:36.800500 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:22:36.815061 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:36.815464 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:36.815549 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:36.829917 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:36.855149 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:36.877131 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 08:22:36.901008 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:36.911159 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:36.935139 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:36.959528 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:37.144068 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 08:22:37.216493 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:37.229991 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:37.234305 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:37.237681 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:37.243252 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:37.282969 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:37.302935 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:37.309010 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 08:22:37.346120 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:37.351758 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:37.360466 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:37.371953 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:37.395906 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:22:37.422585 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:37.436582 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:37.451208 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:37.461406 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:37.465224 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:37.593686 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:37.713858 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:37.717957 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 08:22:37.769554 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:37.772151 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:37.785394 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:37.844704 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:37.853332 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:37.853938 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:37.854393 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:37.988605 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:38.046363 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:22:38.054661 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:38.058121 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:38.064180 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:38.076867 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:22:38.082010 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:38.099082 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:38.112627 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:38.177978 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:38.295200 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 08:22:38.340952 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:38.396199 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:38.434856 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:38.500082 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:38.513314 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:38.526409 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:22:38.541823 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:38.570807 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:38.581393 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:38.602581 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:38.745586 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 08:22:38.746814 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:38.769488 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:38.771212 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:38.778817 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:38.783341 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:38.786098 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:38.827042 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:38.837092 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:38.925544 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:38.937565 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:38.956316 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:39.081681 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:22:39.139676 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:22:39.182355 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:39.187515 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:39.200145 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:39.213280 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:22:39.214313 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:39.336524 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:39.347801 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 08:22:39.347808 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:39.370435 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:39.385150 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:39.409235 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:39.414283 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240501 08:22:39.428735 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:39.432129 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:39.439078 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 106 similar messages]
I20240501 08:22:39.442328 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:39.542032 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 08:22:39.551445 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:39.737350 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:22:39.738328 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:22:39.843175 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:39.852941 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:39.853972 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:39.854748 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:39.884668 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:22:39.884727 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:39.887549 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:39.915472 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:39.973304 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:39.996539 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:22:40.067426 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:40.068574 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:40.080701 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:40.091732 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:40.167248 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:40.187004 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:40.242678 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:40.353498 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:40.354750 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:40.392601 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:40.412959 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:40.444026 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:40.454807 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:40.476296 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:22:40.483584 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:40.491575 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:40.503712 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:40.512166 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:40.573474 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 08:22:40.609823 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:40.620121 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:22:40.760097 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:40.761695 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:40.775523 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:40.779312 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:40.808461 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:40.869920 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:40.940459 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:40.974447 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 08:22:41.009107 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:41.013188 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:22:41.168673 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:41.180809 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:41.203063 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:41.210481 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:41.229619 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:41.262224 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:41.263172 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:41.288087 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:41.344655 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:41.397146 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:41.404250 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:41.405308 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:41.413038 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:41.428108 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:41.450811 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:22:41.465605 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:41.570919 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:41.630383 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:41.673594 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:41.684412 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:41.686902 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 08:22:41.758428 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:22:41.792621 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:22:41.814896 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:41.829197 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:41.830077 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:41.836431 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:41.910531 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 08:22:41.932034 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:41.959345 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:42.034941 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 08:22:42.037748 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:22:42.040158 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:42.050652 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:42.054102 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:22:42.059597 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:42.065454 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:42.213886 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:42.337783 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:42.402583 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:42.434168 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:42.463254 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:22:42.477167 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:42.489356 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:42.490861 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:42.494294 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:22:42.500272 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:42.500690 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:42.506395 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:42.659088 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:22:42.695742 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:42.696065 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:42.735462 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:42.746405 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:42.746754 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:42.795846 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:42.828884 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:42.847728 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:42.851039 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:42.909071 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:42.929364 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:43.027156 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:43.032846 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:43.138983 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:43.158635 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240501 08:22:43.180469 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:43.188561 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:43.192107 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:43.203523 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:43.243759 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:43.376289 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:43.383833 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:43.388727 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:43.395385 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:43.433792 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:43.452411 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:43.488723 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:43.515242 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:43.560715 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:43.582185 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 08:22:43.683036 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:43.685460 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:43.689978 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:22:43.816375 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:43.832445 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:43.844538 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:43.846719 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:43.862573 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 08:22:43.871800 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:43.942629 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:43.942978 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:43.943176 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:43.985210 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:22:44.033879 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:44.040068 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:44.055732 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:44.070497 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:44.121189 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:44.147037 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:44.244612 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 08:22:44.258648 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240501 08:22:44.271674 17980 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:22:44.271854 17980 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:22:44.272480 17980 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:22:44.273751 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:44.273892 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:22:44.273983 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:22:44.274194 17980 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:22:44.284624 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:44.290592 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:22:44.444092 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:44.452986 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:44.464690 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:44.467171 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:44.502014 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:44.504878 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 08:22:44.512187 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:44.531375 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 08:22:44.548945 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:44.558991 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:22:44.643756 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:44.647821 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:44.660224 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:44.691210 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:44.740907 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:44.797639 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:22:44.932678 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:22:44.945325 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:44.954927 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:45.015527 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:45.015981 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:45.031247 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:45.097492 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:45.104662 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:45.104813 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:22:45.105180 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:45.105669 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:45.120957 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:45.160822 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:45.265106 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:45.321287 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:22:45.323693 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:45.324484 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:45.332798 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:45.334611 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:45.347092 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:45.431399 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:45.470988 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:45.485824 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:45.556860 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:45.590406 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:45.604053 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:45.700554 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:45.749830 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:45.762666 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:45.767388 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:45.769403 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:45.860199 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:45.945571 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 08:22:45.948026 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240501 08:22:45.963835 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:45.983826 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:46.001551 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:46.002782 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:46.066895 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:46.124159 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:46.143221 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:46.193768 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:46.203609 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:46.258011 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:46.315179 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:46.370081 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:22:46.398528 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:46.402683 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:46.407073 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:46.407413 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:22:46.419447 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:46.534559 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:46.535107 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:46.583261 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:22:46.584331 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:22:46.612995 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:46.629204 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:46.638381 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:46.644596 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240501 08:22:46.654719 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:46.679088 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:46.695075 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:46.698485 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:46.852509 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:46.868733 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:46.918812 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 08:22:46.979115 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:47.044214 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:47.059303 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:47.078876 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:47.078902 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:47.103787 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:47.159814 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:47.191498 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 08:22:47.208547 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:22:47.216116 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:47.295684 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:47.308105 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:47.309729 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:47.311133 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:47.362972 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:47.480983 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:47.503804 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:47.513422 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:47.563490 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:47.621214 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:47.626443 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:47.740092 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:47.749598 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:47.764665 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:47.764649 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:47.779896 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:47.789587 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:47.902237 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:47.951568 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:47.956332 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:47.959703 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:22:47.961131 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:47.972088 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:47.977658 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:48.029187 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:48.037088 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:48.051960 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:48.195327 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:48.199563 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:48.212596 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:48.272331 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:48.329211 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 08:22:48.352712 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:48.377794 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:48.389864 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:48.393059 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:48.403200 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:48.420771 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 08:22:48.530228 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:22:48.587874 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:48.588192 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:48.591724 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:48.602348 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:48.680608 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:48.723024 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:48.754724 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:48.774066 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:48.844044 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:48.877914 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 08:22:48.878777 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:48.893247 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:48.937369 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:22:48.979660 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:48.997746 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:49.001257 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:49.011900 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:49.020066 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:49.063848 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:49.104076 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:22:49.185348 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:49.191727 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:49.193377 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:49.206063 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:49.211447 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240501 08:22:49.213692 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:49.302819 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:49.456732 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:49.470144 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:49.545213 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 08:22:49.597335 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240501 08:22:49.620380 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:49.626919 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:49.627899 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:22:49.633662 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:49.633953 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:49.647725 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:49.713207 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:22:49.726521 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:49.762573 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:49.765630 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 08:22:49.842283 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:22:49.847059 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:49.847088 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:49.862913 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:49.883184 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:49.889797 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:49.918040 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:50.093345 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:50.146492 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:50.152443 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:50.258708 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:22:50.278723 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:50.289295 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:50.292265 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:22:50.296389 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:50.296428 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:50.360301 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:50.408138 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:50.409515 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:50.414423 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:50.472761 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:50.484704 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:50.485136 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:50.497762 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:50.503012 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:50.543999 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:50.545785 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:50.559664 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:50.563012 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:50.708600 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:50.715415 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 08:22:50.819770 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:22:50.899186 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:50.900430 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:50.901445 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:50.903573 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:50.905978 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:50.983639 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:51.021384 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 08:22:51.028033 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:51.028260 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:22:51.114076 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:51.115799 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:51.129623 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:51.130839 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:51.151453 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:51.254498 18877 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:22:51.254694 18877 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:22:51.255381 18877 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:22:51.256753 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:51.256883 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:22:51.257004 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:22:51.257252 18877 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:22:51.299249 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:51.328347 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:51.401147 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:51.431809 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:51.477888 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:51.498008 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:51.524566 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:22:51.528853 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:51.531936 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:51.533421 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:51.549139 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:51.566862 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:51.580591 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:51.601898 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:22:51.636534 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:51.647802 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:51.740638 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:51.746652 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240501 08:22:51.768352 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:51.777807 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:51.787139 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:51.799726 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:51.800953 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:51.828299 19394 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:22:51.828505 19394 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:22:51.829200 19394 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:22:51.848811 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:51.849046 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:51.849117 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:22:51.849601 19394 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:22:51.907884 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 08:22:51.998195 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:52.047405 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:52.132084 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:52.146031 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:22:52.199076 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:22:52.199128 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:52.212306 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:52.218109 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:52.221910 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:52.277405 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:52.305143 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:52.376338 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240501 08:22:52.397455 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:52.406685 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:52.410480 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:52.415748 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:22:52.416482 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:52.499609 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:52.623929 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:52.674726 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:52.822937 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:52.841849 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:52.846522 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:52.855491 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:52.873361 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:22:52.876971 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:52.909754 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:52.917758 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:52.920450 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:52.949455 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:22:52.994714 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:53.022579 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:22:53.044458 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:53.051038 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:53.052240 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:53.065510 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:53.065639 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:53.073367 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:53.113049 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:53.208307 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:53.248003 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:53.361856 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:53.369904 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 08:22:53.392526 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:53.450279 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:53.475701 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:53.484688 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:53.486992 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:53.490582 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:53.491917 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:53.692413 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:53.697165 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:22:53.702994 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:53.725173 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:53.739338 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240501 08:22:53.745734 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:53.755260 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:53.872785 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:53.920446 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 08:22:53.921265 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:53.926730 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:53.948269 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:53.955696 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:54.019138 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:54.036832 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:22:54.109061 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:54.113447 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:22:54.149397 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:54.154938 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:54.167130 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:22:54.170648 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:54.173065 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:54.203485 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:22:54.342190 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240501 08:22:54.372203 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:54.380884 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:54.395102 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:54.402446 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:22:54.410429 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:54.410744 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:54.455390 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 08:22:54.500051 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:54.527169 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:54.563719 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:54.589555 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 08:22:54.667920 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:22:54.802356 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:54.806797 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:54.812561 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:54.813800 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:54.927263 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:54.945102 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:22:54.955206 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:54.986131 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:55.003793 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:55.005731 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:22:55.010378 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:55.018541 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:55.020756 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:55.051754 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 08:22:55.095099 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:55.116221 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:55.195899 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:55.331333 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:22:55.413187 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:55.416435 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:55.426563 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:55.443439 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:22:55.451493 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:55.502005 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:22:55.587204 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:22:55.603513 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:22:55.606803 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:22:55.625389 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:55.630945 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:55.639287 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:55.646862 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:55.682818 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:55.687645 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:55.718658 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:22:55.742363 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 08:22:55.770869 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:22:55.774067 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:55.851948 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:22:56.007112 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:56.064092 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:22:56.070559 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:56.070967 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:56.082391 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:56.083529 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:56.090924 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 08:22:56.112865 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:56.133890 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:56.198980 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:56.262396 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:22:56.278940 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:56.286845 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:22:56.291823 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:56.296958 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:56.308997 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:56.314986 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:56.362607 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:22:56.401824 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:22:56.452265 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:56.456157 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:22:56.496881 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:56.612042 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240501 08:22:56.709951 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:56.727552 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:56.728514 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:56.734580 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:56.771238 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 08:22:56.772150 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:22:56.782047 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:22:56.802727 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:22:56.891921 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:56.896881 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:22:56.922658 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:56.931170 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:56.934048 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:56.951440 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:56.988148 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:57.008394 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:57.018028 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:57.076053 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 08:22:57.191200 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:22:57.338490 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:57.355953 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:57.370879 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:57.377000 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:57.410815 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:57.430819 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:57.458477 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:57.499567 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:57.500723 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:57.567417 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:57.572156 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240501 08:22:57.574061 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:57.575210 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:57.576882 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:57.577950 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:57.722060 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:57.784623 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:22:57.861932 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:57.884212 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:57.889421 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:57.898468 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:22:58.010016 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:58.011518 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:58.017124 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:22:58.028062 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:58.036376 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:58.038661 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:58.040704 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:22:58.124068 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 08:22:58.178961 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:22:58.243424 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:22:58.244978 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:58.258463 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:58.260404 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:58.279843 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:58.342213 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:58.351372 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:58.396296 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:58.403036 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:58.544911 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:22:58.566365 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 08:22:58.582183 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:22:58.661172 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:58.662227 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:58.681502 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:22:58.682176 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:58.685891 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:22:58.733233 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:58.842401 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:58.843688 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:58.854846 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:22:58.864703 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:22:58.877327 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:58.885886 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:58.888185 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:58.905802 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:22:58.918088 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:58.938585 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 08:22:58.946938 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:58.970331 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:22:58.988080 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:59.177837 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:22:59.217373 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
I20240501 08:22:59.313740 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:59.328636 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:22:59.329072 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:59.334333 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:59.338596 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:22:59.359678 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:59.369401 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 08:22:59.416283 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:22:59.434135 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:59.473223 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:22:59.551339 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:59.561632 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:59.569723 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:22:59.573096 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:22:59.658075 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 08:22:59.797137 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:22:59.805598 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:22:59.849998 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:22:59.870296 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:22:59.893191 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:22:59.974597 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:22:59.977691 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:59.988646 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:22:59.994586 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:22:59.999259 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:00.019364 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:23:00.023238 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:23:00.042878 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:00.050462 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:23:00.203467 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:00.211112 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:00.214174 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:00.233539 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:00.259176 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:00.293834 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:00.312618 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 08:23:00.372946 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:00.421365 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:00.445953 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:00.460116 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:00.544744 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:00.548930 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:23:00.623571 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:00.636090 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:00.645252 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:00.677016 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:00.780311 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:00.811045 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:00.818786 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
I20240501 08:23:00.856062 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:00.862959 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:00.866703 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:00.875651 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:00.876209 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:00.879156 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:00.887171 18877 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:23:00.887447 18877 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:23:00.888520 18877 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:23:00.894100 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:00.902536 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:23:00.902715 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:23:00.903033 18877 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:23:00.950209 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:23:00.973064 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:00.994297 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:23:01.030723 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:01.143261 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 08:23:01.193588 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:01.222076 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:23:01.246677 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:01.280925 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:01.295800 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:01.300859 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:01.303040 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:01.316298 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:01.339538 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:01.340636 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:01.447607 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:01.481829 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:23:01.482057 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:23:01.490622 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:01.501771 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:01.502925 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:01.503681 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:01.531910 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:01.552155 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:01.761287 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 08:23:01.778717 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:23:01.779595 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:01.786286 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:01.813424 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 08:23:01.915591 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:01.918032 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:01.918423 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:23:01.954766 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:01.961712 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:01.968422 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:02.028795 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:02.140250 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:02.190003 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:02.194572 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:02.199755 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:02.213013 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:02.248858 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:23:02.256847 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:23:02.308840 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:02.391368 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:23:02.424691 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:02.482343 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:23:02.583770 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:23:02.600749 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:02.608212 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:02.612840 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:02.614197 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:02.685763 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:02.816877 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:02.825274 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:02.832149 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:23:02.840076 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:02.842494 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:02.857761 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:02.863206 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:02.945632 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 08:23:02.991420 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:03.006141 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:03.012387 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:03.210739 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:23:03.216214 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:23:03.230474 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:03.242462 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:03.251410 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:03.255050 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:03.288246 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:03.305660 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 08:23:03.309295 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:03.392536 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:03.392704 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:03.434826 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:03.444031 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:03.446449 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:03.458762 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:03.470886 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:03.483887 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:03.575416 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:03.583329 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:03.615486 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 08:23:03.727550 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:23:03.836050 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:03.853773 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:23:03.855386 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:23:03.856948 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:03.865877 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:03.867575 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:03.881105 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:03.942106 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240501 08:23:03.950830 20742 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:23:03.951045 20742 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:23:03.951731 20742 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:23:03.954102 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:03.954952 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:03.955147 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:03.955260 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:23:03.955519 20742 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:23:03.969620 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:03.987427 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:23:04.046512 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:04.049204 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:04.052327 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:23:04.068467 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:04.070835 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:04.072024 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:04.219307 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:04.262854 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 08:23:04.283362 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:04.324478 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:04.326066 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 08:23:04.350759 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:23:04.496928 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:04.500766 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:04.502827 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:04.510974 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:04.515638 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:04.550249 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:04.571339 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:04.608705 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:04.641816 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:04.671877 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240501 08:23:04.688702 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:04.700611 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:23:04.723668 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:04.752545 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:04.758909 6484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20240501 08:23:04.767477 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:04.792938 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9b9bd7500e8747c6b8ee94a1faa1c5f7: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:23:04.797338 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:04.798131 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:05.021589 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 08:23:05.056233 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:05.078859 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:05.144362 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:05.196372 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:05.211278 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:05.225574 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:05.234699 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:05.254664 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:05.372588 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:05.377535 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
I20240501 08:23:05.400919 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:05.403905 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:05.420030 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:05.438380 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:05.443953 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:05.494608 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:05.524114 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:05.534865 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:05.596473 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:05.619077 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:05.698405 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:05.705891 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:05.750444 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:05.810925 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:05.824767 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:05.838613 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:05.842118 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:05.843417 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:05.894383 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:05.917695 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 08:23:06.019026 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:06.019544 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240501 08:23:06.028622 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:06.038992 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:06.043546 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:23:06.048101 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:06.066617 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:23:06.066993 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:06.107362 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:06.222961 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:06.223134 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:06.233906 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:06.244930 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:06.245076 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:06.267442 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d7903106d7e74e1ea102b1abdbfce63e: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:23:06.448822 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:06.461580 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:06.473814 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:06.475008 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:06.539356 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:06.566701 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:06.605202 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:06.621265 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:06.629258 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:23:06.634631 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:06.644318 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:06.644944 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:06.655367 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:06.666484 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:06.691798 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:06.718783 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 08:23:06.830157 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:23:06.845239 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:06.889376 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 08:23:06.899519 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:06.925611 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:07.025805 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:23:07.045296 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:07.073755 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:07.074785 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:07.082863 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:07.102414 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:07.114589 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:07.127667 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:07.135962 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:23:07.235009 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:07.282157 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:07.296483 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:07.302918 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:07.306066 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:07.566722 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:07.646466 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:07.656284 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:07.689580 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:07.694962 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:07.705458 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:07.706043 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:07.774012 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:07.776548 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 193de3c11ae54408a64ae3007bf88920: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:07.822227 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 08:23:07.832628 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:23:07.891594 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:07.907745 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:23:07.919731 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:07.922879 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:07.931672 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:23:07.955035 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:08.047269 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:08.069573 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:08.118065 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:08.125511 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:08.169526 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:08.184226 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:08.240543 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:08.304806 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:23:08.318377 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:08.339186 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:08.343199 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:08.359695 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:08.410276 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 08:23:08.440739 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:08.530462 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:08.530592 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:08.538054 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:08.545053 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:08.570554 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:08.574950 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:08.611826 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:08.645428 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:08.680172 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:08.694419 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 08:23:08.716692 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:08.794314 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:08.800912 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:08.935293 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:23:08.943224 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:08.951328 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:08.951961 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:08.955691 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:08.960209 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:23:08.966225 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:08.971292 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:08.976038 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 08:23:08.977085 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:09.127115 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:09.136781 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:23:09.154587 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240501 08:23:09.156574 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:09.163070 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:09.163551 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:09.200903 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:09.215690 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:09.231742 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:09.364218 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:09.469159 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:09.487653 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:23:09.566854 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:09.582130 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:09.583401 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 08:23:09.589011 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:23:09.590417 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:09.590999 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:09.591557 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:09.662916 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:09.713212 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240501 08:23:09.716022 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:09.750363 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:09.781080 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:09.787112 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:09.790968 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:09.793143 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:10.014292 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:10.027051 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:23:10.133265 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:23:10.182719 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:10.198092 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:10.215145 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:10.217890 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:10.299897 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:10.303907 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:10.306034 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 08:23:10.322536 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:10.369568 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:23:10.373098 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:10.384981 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:10.387029 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:10.394918 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:10.496429 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:10.533144 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:10.533309 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:10.565886 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:10.631973 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:10.658030 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:10.727965 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:10.781656 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:23:10.794596 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:10.799512 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:10.805094 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:10.821319 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:10.859881 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 08:23:10.875277 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:10.879601 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:10.916950 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:10.937350 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:10.972162 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:23:10.982069 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:10.986698 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:10.994522 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:11.024586 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:11.095135 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 08:23:11.200925 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:11.262986 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:11.299636 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:11.322912 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:11.334750 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:11.402374 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:11.412587 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:11.413619 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:11.418506 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:11.454730 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:11.467797 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:11.532663 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:11.560814 20999 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:23:11.561028 20999 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:23:11.561725 20999 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:23:11.567405 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:11.567610 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:11.567682 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
W20240501 08:23:11.568073 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240501 08:23:11.568284 20999 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:23:11.597698 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:11.601819 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:11.602559 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:11.610357 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:11.621026 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:11.624362 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:11.688091 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:11.786118 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:11.820827 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:11.831730 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:11.835038 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:23:11.853157 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:11.861794 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:23:11.972977 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:23:11.992815 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:11.996537 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:11.999071 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:12.005661 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:12.018342 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:12.020686 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:12.064157 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:12.088330 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:12.117123 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240501 08:23:12.186353 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:12.190230 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:12.193581 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:12.197604 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:12.309170 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:12.438814 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:12.597143 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:12.604501 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:12.606428 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:12.622501 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:12.639083 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:12.742560 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:12.744505 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:23:12.811815 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:12.822517 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:12.827608 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:12.827978 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:12.839231 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:12.839800 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:12.891374 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:12.909291 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:12.931771 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:12.959082 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 08:23:13.027406 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:13.057317 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:13.097975 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:13.202453 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:13.212384 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:13.230372 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:13.230986 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:13.232082 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:13.239045 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:23:13.248605 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:13.274533 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:13.276839 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:13.385654 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:13.410493 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:13.426632 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:13.434010 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:13.447867 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:13.475937 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:13.503513 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 08:23:13.568447 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:13.570559 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:23:13.646564 21493 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:23:13.646749 21493 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:23:13.647374 21493 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:23:13.648532 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:13.648689 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:23:13.648912 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:23:13.649206 21493 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:23:13.678578 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:13.719465 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 08:23:13.801594 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:13.804025 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:23:13.826282 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:13.838454 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:13.844588 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:13.847966 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:13.861176 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:13.915822 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:13.933550 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:13.980784 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240501 08:23:14.013978 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:14.022619 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:23:14.041277 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:14.044306 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:14.044472 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:14.061692 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:14.072363 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:14.175161 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:14.386663 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:14.391078 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:14.395457 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:14.411954 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:14.419013 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:14.437240 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:14.439224 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:23:14.472491 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:14.514766 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 08:23:14.516623 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:23:14.527456 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 08:23:14.534768 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:14.570797 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:14.632683 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:14.634061 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:23:14.655757 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:14.656113 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:14.656611 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:14.656898 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:14.675385 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:14.730098 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 08:23:14.801841 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:14.973276 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:15.066550 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:15.067121 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:15.067853 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:15.093317 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:15.098431 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:23:15.224849 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:15.258702 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:23:15.267158 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:15.270553 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:15.273124 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:15.276175 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:15.284302 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:15.392052 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 08:23:15.403596 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:15.405087 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:15.421598 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:15.463446 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:15.489403 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:15.545584 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:15.555816 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:23:15.572573 22236 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:23:15.572757 22236 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:23:15.573390 22236 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:23:15.574805 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:15.575013 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:15.575124 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:23:15.575357 22236 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:23:15.627852 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:15.653687 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:23:15.662083 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:15.678794 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:15.683362 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:15.690126 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:15.707801 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:15.757084 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:15.806337 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:15.821239 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:15.849169 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:15.867548 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:15.870941 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:15.878755 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:15.884475 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240501 08:23:15.887192 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:15.927992 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:15.935742 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:16.142300 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 08:23:16.182067 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:16.232939 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:16.237658 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:16.300871 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:16.307219 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:16.313398 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:16.318940 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:16.322117 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:16.343930 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:16.368566 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:16.415787 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:23:16.483533 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:16.494361 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:16.499388 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:16.500635 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:16.583029 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:16.615017 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:16.624837 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:16.801735 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:16.889614 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:16.890225 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:16.893816 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:23:16.907209 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:16.908125 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:16.921739 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:17.016505 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 08:23:17.017318 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:17.060866 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:17.065161 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:23:17.074280 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:17.083591 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:17.094583 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:17.100461 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
I20240501 08:23:17.103816 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:17.106750 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:17.123602 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:23:17.205533 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:17.205546 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 08:23:17.278975 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:17.294765 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:17.423458 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:17.487982 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:17.493732 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:17.512624 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:17.513805 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:17.554841 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:17.708513 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:17.726576 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:17.731823 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:17.732620 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:17.751744 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:17.752339 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:17.753784 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:17.789559 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:17.860232 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:17.875654 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:17.911326 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:17.917872 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:17.945879 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 08:23:17.972551 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:18.030304 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:18.108414 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:23:18.144901 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:18.159538 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:18.160531 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:18.164378 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:23:18.171437 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:18.184558 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:18.210132 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:18.259807 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
I20240501 08:23:18.325403 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:18.325919 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:18.332190 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:23:18.339920 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:18.341559 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:18.353737 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:18.361930 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:18.445137 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:18.454056 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:18.516923 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:18.588632 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
I20240501 08:23:18.750121 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:18.754664 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:18.767593 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:18.781072 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:18.845546 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:18.868857 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:18.875557 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:18.959206 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:23:18.960554 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:18.966603 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:18.974517 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:18.976195 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:18.999219 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:19.004151 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 08:23:19.079186 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:19.093132 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:19.116722 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:23:19.347324 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:19.357810 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:19.368547 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:19.372794 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:19.388495 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:19.423806 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:19.494349 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:19.514019 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:23:19.541465 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:19.544945 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:19.549856 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:19.557317 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:23:19.566036 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:23:19.581935 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:19.585096 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:19.628373 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 08:23:19.630890 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:19.647720 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:19.723538 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:19.772794 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 08:23:19.790167 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 08:23:19.872812 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:19.952853 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:19.960148 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:19.966244 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:19.978881 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:19.982195 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:19.985092 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:20.091845 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:20.155818 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:23:20.167198 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:20.168521 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:20.180159 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:23:20.197227 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:20.206050 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:20.322988 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:20.332610 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:20.368403 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:20.402536 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:20.405412 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:20.432425 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:20.490614 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:20.499894 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 08:23:20.570530 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:20.572580 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:20.595335 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:20.599536 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:20.622869 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:20.642298 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:20.680070 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:20.736016 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:20.792840 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:20.809343 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:20.811050 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:20.811550 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:20.813283 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:20.818323 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:20.818885 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:20.845986 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:20.879093 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 08:23:20.969722 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 08:23:20.978649 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:21.098104 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:21.195174 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:21.195856 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:23:21.218588 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:21.232101 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:21.236994 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:21.240967 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:21.259887 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:21.336900 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:21.370810 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:21.390938 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:21.421111 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:21.421540 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:21.428532 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:21.448992 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:21.607697 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:21.637053 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:21.699663 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:21.832360 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:21.834095 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:23:21.842923 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:21.845928 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:21.851140 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:21.856797 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:21.994019 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:22.022684 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:22.035071 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:22.035140 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:22.040374 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:22.048343 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:22.079313 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:23:22.080247 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:22.090111 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:22.198312 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:22.225621 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 08:23:22.235193 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:22.248255 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:23:22.260149 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:22.301098 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 08:23:22.315912 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:22.422080 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:22.425100 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:22.439606 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:22.441979 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:22.490480 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:22.528896 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:23:22.628046 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:22.632431 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:22.632908 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
I20240501 08:23:22.648190 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:22.649108 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:22.653483 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:22.699469 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:22.729887 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:22.832260 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:22.879693 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:22.887113 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:22.931563 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:22.957273 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:22.969733 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:23:22.995313 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:23:23.043116 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:23.043769 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:23.049065 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:23.055454 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:23.122907 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:23.213886 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:23.222694 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:23.227839 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:23.239632 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:23:23.240576 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:23.292203 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:23.454301 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:23.481662 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:23.496232 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:23.499119 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:23.530236 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:23.541690 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 08:23:23.566876 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:23.597915 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 08:23:23.607339 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:23:23.611148 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:23.615511 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:23.629647 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:23.643285 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:23.715514 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:23.771436 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:23:23.847841 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:23.854148 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:23.860188 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:23.866856 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:23.872586 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:23.961519 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:23.998742 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:24.030366 6571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 08:23:24.031134 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:24.052284 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:24.199064 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:24.221333 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:23:24.255514 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:24.258430 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:24.272130 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:24.284169 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:24.359712 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:23:24.447297 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:24.449927 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:24.459445 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:24.463979 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:24.564879 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:24.578037 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:24.667912 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:24.709308 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:24.717989 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:24.750460 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:24.787882 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 08:23:24.792658 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:24.801946 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:24.836409 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 08:23:24.837467 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:23:24.839011 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:24.848764 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:24.857262 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:24.870630 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:24.871992 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:25.007937 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:25.026002 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:25.033600 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:25.045668 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
I20240501 08:23:25.046813 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:25.047541 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:25.058790 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:25.059808 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:25.185056 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:23:25.209642 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:25.218752 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:25.266534 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 08:23:25.411798 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:25.416343 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:23:25.418999 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:25.427660 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:25.432740 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:25.438228 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:25.598409 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:25.614820 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:23:25.621317 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:25.621433 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:25.635493 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:25.640487 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:23:25.641732 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:25.652976 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:25.704988 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:25.720237 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:25.829226 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:25.923907 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:25.939592 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:25.942684 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:25.992755 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
I20240501 08:23:26.049885 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:26.054286 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:26.055356 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:26.063557 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:26.110818 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:26.162951 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:26.228507 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:26.229991 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:26.236333 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:26.239919 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:26.241036 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:26.275398 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 08:23:26.283967 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:26.318622 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:26.353021 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:26.373281 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:26.478516 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:23:26.631834 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:26.638104 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:23:26.639214 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:26.647636 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:26.654217 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:26.685644 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:26.706820 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:26.823704 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:26.827548 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:26.838455 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:23:26.853592 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:26.853610 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:26.879012 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:26.933547 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:26.985954 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:26.989074 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:23:26.994510 23442 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:23:26.994706 23442 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:23:26.995417 23442 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:23:26.996683 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:26.996954 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:23:26.997064 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:23:26.997299 23442 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:23:27.078541 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:27.105461 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:27.158918 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 08:23:27.188720 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:27.211939 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:23:27.230213 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:27.256175 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:27.256999 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:27.257779 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:27.306102 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:27.354305 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:23:27.384815 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:23:27.416003 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:27.426396 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:23:27.430670 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:27.437503 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:27.441397 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:27.442121 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:27.461714 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:27.539700 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:27.548058 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:27.778667 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 08:23:27.784716 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:27.817406 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:27.834916 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:27.835870 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:27.851025 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:27.893671 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:27.932299 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:27.966131 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:28.011159 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:28.050119 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:23:28.062119 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:28.063227 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
I20240501 08:23:28.074566 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:28.078044 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:28.087793 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:28.108217 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:28.152472 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:28.159114 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:28.163774 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:28.185655 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:28.295816 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20240501 08:23:28.477857 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:28.480160 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:28.485074 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:28.500253 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:28.508170 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:28.532562 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:28.541458 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:28.582935 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:28.608353 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 08:23:28.633618 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:28.641515 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:28.659072 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:23:28.709406 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:28.724210 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:28.734385 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:28.735003 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:28.769174 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:28.821971 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:28.910871 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:29.071949 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240501 08:23:29.119184 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:29.135852 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:29.138478 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:29.151484 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:29.156240 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:29.172562 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:29.195006 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:29.237315 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:29.244956 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:29.305004 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:23:29.341609 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:29.352340 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:29.356509 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:29.366546 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:29.471220 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240501 08:23:29.502676 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:29.540679 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:29.548826 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:23:29.585516 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:29.623761 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:29.624612 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:29.675426 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:23:29.750564 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:29.756906 23824 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:23:29.757079 23824 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:23:29.757716 23824 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:23:29.762231 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:29.762461 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:29.762533 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:23:29.762979 23824 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:23:29.767827 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:29.774950 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:23:29.775549 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:29.777514 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:29.822351 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:29.846846 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240501 08:23:29.930972 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:23:29.951864 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:29.956501 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:29.962203 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:29.964650 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:23:29.973022 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:29.973104 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:30.013109 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:30.079810 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:30.149092 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:30.213409 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:30.213464 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:23:30.216252 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240501 08:23:30.289019 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240501 08:23:30.364813 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:23:30.390477 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:30.401840 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:30.402635 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:30.409341 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:30.453405 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:30.559844 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:30.564258 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:30.581626 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:30.590555 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:30.601289 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:30.607198 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:30.608777 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:30.631871 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:30.632623 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:30.676743 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:30.803828 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:30.898706 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240501 08:23:30.909965 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240501 08:23:30.971545 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:23:31.002912 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:31.003439 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:31.034238 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:31.061461 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:31.063222 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:31.094174 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:31.127799 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:31.166548 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:31.188544 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:31.193397 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:31.209535 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:23:31.217780 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:31.232826 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:31.242775 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:31.243386 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:31.281198 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:23:31.646968 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:31.649714 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:31.660455 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:31.661660 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:31.679090 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:31.689759 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:31.740561 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:31.743834 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:31.780239 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:31.817029 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:23:31.847465 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:31.853498 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:31.858208 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:31.867367 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:31.940557 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:31.946115 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240501 08:23:32.115758 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:32.118531 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:32.121873 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:23:32.145165 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:32.152863 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:32.154678 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:32.156770 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:32.205103 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:32.247033 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:32.248924 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:23:32.259923 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:32.268219 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:32.280967 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:32.317370 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:32.409163 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 74 similar messages]
I20240501 08:23:32.459411 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:32.475204 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:32.482110 23442 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:23:32.482331 23442 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:23:32.483050 23442 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:23:32.485646 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:32.492048 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:32.502775 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:23:32.503089 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:23:32.503413 23442 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:23:32.521379 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:32.537683 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240501 08:23:32.615025 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:32.702592 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:32.727697 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240501 08:23:32.728262 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:32.745127 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:32.815716 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:32.844645 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:32.850616 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:23:32.888893 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:32.892961 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:32.893210 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:23:32.898458 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:32.907544 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:32.975507 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:33.026611 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:33.077316 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:23:33.085299 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:33.085764 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:33.103222 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:33.104182 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:33.110716 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:33.167917 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:33.231889 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:33.258610 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240501 08:23:33.356457 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:33.422689 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:33.446134 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:23:33.488196 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:33.503795 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:33.512712 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:23:33.531476 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:33.534283 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:33.538544 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:33.546232 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240501 08:23:33.638094 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:33.677170 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:33.708627 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:33.719533 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:33.737845 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:33.740376 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:33.766932 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:33.830555 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:33.877846 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:34.024183 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:34.115628 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240501 08:23:34.131579 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:34.133090 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:23:34.153985 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:34.155537 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:34.163494 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:34.175976 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:34.186959 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:34.227941 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:23:34.343966 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:34.358819 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:34.367650 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:34.368692 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:34.371555 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:34.421878 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:34.510241 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:34.533485 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:23:34.539247 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240501 08:23:34.593703 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:34.699684 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:34.729401 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:34.748921 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:23:34.762408 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:34.774107 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:34.775220 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:34.783970 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:34.817622 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:34.835336 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:34.847754 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:34.874390 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240501 08:23:34.944227 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:23:34.976408 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:34.976784 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:34.993348 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:34.997026 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:34.999325 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:35.015062 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:35.056687 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240501 08:23:35.131485 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240501 08:23:35.172073 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:35.282667 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:35.343401 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:23:35.373111 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:35.375713 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:23:35.380188 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:23:35.383720 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:35.402701 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:35.403868 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:35.417614 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:35.446985 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:35.452333 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:35.472252 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:35.489948 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:35.495955 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:35.584703 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:35.585579 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:35.585943 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:35.597719 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:35.642298 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:35.706311 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240501 08:23:35.775581 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:35.797374 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:35.860265 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:23:35.981056 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:35.996845 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
I20240501 08:23:35.996862 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:36.005088 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:36.018833 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:23:36.021605 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:36.056623 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:36.069314 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:36.174183 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:23:36.196216 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:36.210600 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:36.211903 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:23:36.211948 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:36.217797 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:36.231835 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:36.302392 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:36.348287 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:36.429667 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:23:36.619124 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:36.631350 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:36.632653 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:36.637032 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:23:36.638656 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:36.667147 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:36.671403 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:36.746362 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:36.786857 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:36.787420 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:23:36.837517 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:36.839278 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:36.847471 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:36.852602 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:36.918987 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:36.924679 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:23:36.933591 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:36.997002 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:37.055728 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:37.070333 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240501 08:23:37.214233 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:37.244758 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:37.251632 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:37.255695 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:37.262560 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:37.288012 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:37.344223 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:37.355245 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240501 08:23:37.418735 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:23:37.422448 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:37.427115 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:37.433593 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:37.436087 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:23:37.439172 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:37.478608 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:37.495241 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:37.632437 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:37.655109 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:37.657929 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:37.716688 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:37.724983 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:37.730180 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:37.734639 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
I20240501 08:23:37.822206 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:37.828676 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:37.833891 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:37.843868 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:37.871634 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:37.931385 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:37.980909 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:38.025097 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:38.025497 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:38.041546 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:38.046468 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:38.109555 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:38.116835 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:38.187232 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240501 08:23:38.200395 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:38.288604 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:38.357074 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:38.398787 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:23:38.409873 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:38.414687 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:38.415992 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:38.418642 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:38.477664 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:38.497614 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:38.539917 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:38.559082 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:38.567751 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:23:38.593817 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:38.602602 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:38.607894 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:38.622154 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:38.654105 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240501 08:23:38.750283 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:38.823379 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:38.941551 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:39.013950 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:39.018373 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:39.026916 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:39.044584 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:39.059197 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:39.060096 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:39.073413 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:39.074668 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:39.145804 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:39.191959 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 93 similar messages]
I20240501 08:23:39.234247 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:39.235029 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:39.244693 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:39.245900 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:39.297222 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:39.377954 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:23:39.404589 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:39.435488 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:39.463945 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:39.545171 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:39.615661 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
I20240501 08:23:39.624459 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:39.633934 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:39.639537 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:39.643469 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:39.653360 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:39.679047 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:39.698074 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:39.731699 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
I20240501 08:23:39.851334 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:39.851509 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:39.855954 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:23:39.859140 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:39.864670 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:39.875209 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:39.881990 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:39.967279 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240501 08:23:39.977232 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:40.098299 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:40.169807 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:40.173856 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:40.200456 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:23:40.239064 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:40.256062 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:40.260285 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:40.279657 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:40.283214 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:40.342114 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:40.352200 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240501 08:23:40.398399 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:40.442989 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:23:40.444603 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:40.455430 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:40.466488 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:40.467054 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:40.507725 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:40.614212 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240501 08:23:40.653935 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:40.776013 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:40.834744 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:23:40.861301 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:40.863431 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:40.865885 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:40.880100 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:40.881055 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:40.898057 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:40.909629 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:40.940182 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:40.943051 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:40.992985 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:23:41.080344 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:41.083338 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:41.092010 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:41.096724 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:41.122695 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:41.142060 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:41.268716 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:41.283169 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:41.332572 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240501 08:23:41.401268 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:23:41.474440 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:41.486593 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:41.495893 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:23:41.497902 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:41.527063 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:41.528801 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:41.529572 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:41.611503 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:41.680797 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:41.697422 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:23:41.733273 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:41.743075 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:41.755374 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:41.767694 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:23:41.769268 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:41.782101 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:41.812731 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240501 08:23:41.904747 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:41.941136 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:41.973695 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:41.981742 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240501 08:23:41.981742 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:23:42.078047 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:42.161721 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:23:42.182853 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:42.208406 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:42.209332 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:42.211297 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:42.313905 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:42.318229 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:42.331286 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:42.386060 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:42.387730 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:42.393771 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:42.407050 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:42.462875 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:23:42.503019 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:42.579731 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:42.592857 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:42.609560 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:42.621788 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240501 08:23:42.647627 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240501 08:23:42.723002 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:42.740550 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:42.748805 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:23:42.792197 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:42.792881 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:42.803217 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:42.812897 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:42.838003 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:42.991739 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:23:43.009240 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:43.012444 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:43.019845 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:43.029129 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:43.081732 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:43.151819 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:43.211411 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:43.214787 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:43.220085 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:43.339568 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:23:43.402947 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:43.403241 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:43.411191 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:43.415500 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:43.422559 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:43.519091 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:23:43.531803 24179 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:23:43.531981 24179 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:23:43.532620 24179 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:23:43.534296 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:43.534482 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:43.534555 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:23:43.535009 24179 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:23:43.599601 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:43.607882 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:43.613328 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:43.617373 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:43.627660 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:43.643923 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:43.661412 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:43.745954 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:43.755882 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:43.814330 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:43.817025 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240501 08:23:43.845386 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:43.964424 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:43.970918 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:44.006353 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:44.022186 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:44.029188 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:44.049639 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:44.056203 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:44.089130 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:44.183416 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:44.203076 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240501 08:23:44.203925 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:44.234566 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:44.242102 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:44.249543 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:44.261014 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:44.273530 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:44.382416 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:44.478155 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:44.491523 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:23:44.577032 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240501 08:23:44.613911 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:23:44.649147 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:44.657611 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:44.667694 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:44.668763 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:44.677774 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
I20240501 08:23:44.681277 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:44.696565 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:23:44.847012 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:44.848811 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:23:44.855284 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:44.855494 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:44.864778 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:44.869591 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:44.871114 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:44.885215 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240501 08:23:44.918063 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:44.944334 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:44.969102 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 08:23:45.063633 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:45.071050 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:45.096076 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:45.254115 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:23:45.268901 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:45.275012 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:45.283185 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:45.283851 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:45.416376 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:45.434576 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:45.453528 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:45.460471 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:45.462018 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:45.475415 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:45.509523 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:23:45.510360 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240501 08:23:45.548066 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:45.644129 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:45.648823 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:45.674336 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:45.784938 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:45.802561 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:23:45.846525 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:45.857170 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:45.857750 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:45.862182 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:45.929558 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:45.961905 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:46.007489 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:46.048386 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:46.060813 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:46.068559 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:46.068989 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:46.152829 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:46.188406 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240501 08:23:46.194175 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:46.209193 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:46.214942 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:46.320780 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:46.336839 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:46.355868 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:46.409802 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:46.424412 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:46.436544 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:46.438576 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:46.443678 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:46.458478 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:46.459054 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:46.537235 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:46.546124 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:23:46.675729 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:46.682998 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:46.683429 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:46.685546 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:46.703410 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:46.786487 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:46.860953 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:46.880440 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:46.889679 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:46.919365 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:46.948168 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:23:47.080976 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:47.093410 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:47.104146 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:47.110308 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:47.117226 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:47.254032 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:47.262537 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:47.272821 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:23:47.297641 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:23:47.298478 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:47.310331 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:47.312315 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:47.312463 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:47.326215 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240501 08:23:47.333711 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:47.341217 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240501 08:23:47.425520 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240501 08:23:47.471418 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:23:47.490373 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:47.555290 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:47.614480 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:47.670083 26138 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:23:47.670322 26138 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:23:47.671113 26138 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:23:47.678630 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:23:47.679100 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:47.679229 6702 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:23:47.679517 26138 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:23:47.707376 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:47.725091 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:47.727200 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:23:47.733898 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:47.746730 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:47.794628 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:47.821378 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:47.911859 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:47.934098 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:47.942462 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:47.949589 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:47.993297 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:47.995473 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:48.045207 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:48.234650 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:48.254894 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:48.256516 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:48.278563 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:48.278882 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240501 08:23:48.336937 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:48.344516 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:48.354610 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:48.356649 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:23:48.357910 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:48.407027 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:48.415395 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:48.425240 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:48.487622 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:23:48.517911 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:48.530557 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:48.543569 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:48.545055 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:48.591878 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:48.592067 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:48.624517 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240501 08:23:48.807891 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:48.855593 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:48.881187 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:48.885682 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:23:48.938243 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:48.941895 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:48.966361 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:48.967614 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:48.968969 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:48.986994 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:48.992837 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:49.063369 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:23:49.140430 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:49.154387 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:49.190289 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:49.202139 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:23:49.214944 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:49.241478 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 08:23:49.332026 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:49.353240 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:49.389030 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:49.394428 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:49.412794 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:49.419366 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:49.460736 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:23:49.595757 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:49.607740 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:49.611052 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240501 08:23:49.614629 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:49.626215 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:49.694654 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:49.741072 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:23:49.744640 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:23:49.792414 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:49.797660 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:49.803121 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:23:49.805217 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:49.809746 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:49.841897 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:49.855419 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:49.874442 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240501 08:23:49.876065 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:49.972551 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:49.997325 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240501 08:23:50.060567 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 08:23:50.206321 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:23:50.207814 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:50.208750 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:50.218160 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:50.224174 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:50.267122 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:50.269969 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:50.391130 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:50.402595 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:50.403998 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:50.423767 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:50.436702 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:50.521397 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:50.529520 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:50.603324 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:23:50.639073 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240501 08:23:50.761498 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:50.806419 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:50.810413 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:50.812758 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:23:50.824128 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:23:50.826910 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:50.830961 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:50.840384 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:50.901031 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:50.936859 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:50.959863 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:51.009585 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:51.016147 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:51.019940 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:51.027104 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:23:51.030309 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:51.044486 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:51.121325 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240501 08:23:51.143292 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:51.235738 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:51.248286 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:51.275468 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:51.342778 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:51.392238 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:51.413283 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:23:51.431785 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:51.441358 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:51.442430 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:51.447873 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:51.508203 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:51.526166 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:51.564836 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:23:51.606974 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:51.610530 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:51.620872 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:51.640347 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:51.757411 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:51.796685 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:23:51.808338 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:51.834041 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:51.860968 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:51.904476 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:51.905226 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:51.987604 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:52.008773 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:23:52.042043 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:52.049533 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:52.054654 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:23:52.082551 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:52.085798 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:52.145700 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240501 08:23:52.216291 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:52.247645 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:52.249094 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20240501 08:23:52.249461 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:52.269946 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:52.288770 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240501 08:23:52.295629 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:23:52.303398 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:23:52.319104 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:52.329449 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:52.466325 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:52.526115 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:52.556507 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:23:52.593452 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:52.645035 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:23:52.690289 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:52.693070 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:52.713210 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:52.722337 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:52.725654 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:52.798873 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:23:52.882732 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:52.896332 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:52.901168 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:52.903779 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:52.948695 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:23:52.973676 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:53.031517 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:53.129029 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:53.208730 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240501 08:23:53.256096 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:23:53.292896 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:53.294461 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:53.301721 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:53.309471 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:53.328980 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:23:53.351200 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:23:53.354750 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:23:53.479308 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:53.488716 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:53.495857 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:53.510661 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:53.514765 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:23:53.517619 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:53.558543 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:53.563015 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:53.575971 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:53.628166 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:53.689369 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240501 08:23:53.819872 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:23:53.851819 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:23:53.893222 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:53.895448 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:53.899020 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:53.905563 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:53.930176 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:53.942292 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:23:53.945351 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:54.013451 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:54.040175 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:54.081940 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:54.088887 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:54.108920 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:54.112321 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:54.113106 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:54.136358 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:23:54.206092 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:23:54.260982 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 08:23:54.335732 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:54.379186 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:54.380259 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:23:54.475517 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:54.525199 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:54.525699 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:54.539434 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:54.542677 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:54.546816 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240501 08:23:54.550796 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:54.561064 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:23:54.642010 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:23:54.647336 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:54.651227 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:54.734776 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240501 08:23:54.735180 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:54.738649 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:54.762789 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:54.763589 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:54.815174 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240501 08:23:54.919072 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:54.978380 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:54.981645 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:54.994128 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:23:55.000535 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:55.045532 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:55.055651 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:55.061718 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:23:55.072607 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 107 similar messages]
I20240501 08:23:55.154449 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:55.171147 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:55.181531 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:55.190667 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:55.357722 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:55.374603 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:55.380818 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:55.393877 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:55.459475 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:55.602559 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:55.621120 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:55.671463 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:23:55.672349 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240501 08:23:55.695554 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:23:55.745242 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:23:55.772624 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:55.779356 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:55.795665 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:55.800804 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:55.837621 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:23:55.852280 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:55.943692 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:23:55.951404 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:23:55.982774 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:55.995317 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:56.003871 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:23:56.004724 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:56.019902 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:56.028019 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240501 08:23:56.061489 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:56.184199 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:56.216533 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:56.226099 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:56.360502 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:23:56.381564 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:23:56.396457 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:56.408834 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:56.414389 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:56.429559 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:23:56.431473 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:56.448323 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:56.453457 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:56.496978 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:56.597379 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:56.604383 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:56.616695 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:56.621795 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:56.622370 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:23:56.634826 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:56.719705 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:23:56.776053 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:56.797571 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:23:56.826512 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:56.938791 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:57.026278 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:23:57.030496 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:57.037976 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
I20240501 08:23:57.046187 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:57.047161 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:57.054816 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:57.067597 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:23:57.165598 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:57.169528 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:23:57.231622 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:57.233820 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:57.239316 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:57.241919 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:57.265177 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:57.347672 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240501 08:23:57.395231 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:57.395318 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:23:57.432909 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240501 08:23:57.470420 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:57.499857 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:57.567780 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:23:57.629757 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:57.638517 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:23:57.665855 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:57.667299 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:57.674109 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:57.698819 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:57.748823 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:23:57.804220 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:23:57.881919 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:57.888587 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:57.896559 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:57.906455 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:57.909804 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:57.935350 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240501 08:23:57.952700 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:58.038038 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:58.054677 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:23:58.174801 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:23:58.250337 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:23:58.283407 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:58.291415 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:58.300221 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:58.307399 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:58.338138 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:58.404923 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:58.443969 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:23:58.459219 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:23:58.465071 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:58.469154 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:58.481245 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:58.483470 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:58.484395 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:58.491647 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:23:58.561084 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:23:58.574066 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240501 08:23:58.637154 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:58.789407 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:58.809713 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:23:58.833343 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:23:58.847627 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:58.860854 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:58.862628 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:58.862993 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:23:58.864392 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:23:58.868784 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:58.923669 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:23:58.936630 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:23:59.045320 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:59.052565 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:59.055938 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:23:59.063722 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:23:59.071776 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:59.199314 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:59.266841 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 08:23:59.291091 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:23:59.314471 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:59.381289 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:23:59.406574 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:23:59.430524 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:59.441076 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:59.456563 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:59.468817 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:59.469069 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:23:59.485491 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:23:59.597884 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240501 08:23:59.674810 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:23:59.677706 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:23:59.679112 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:23:59.681316 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:23:59.713085 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:23:59.763593 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:23:59.793706 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240501 08:23:59.799851 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:23:59.868506 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:59.904268 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:23:59.957425 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:00.002324 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:24:00.059585 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:00.063660 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240501 08:24:00.092116 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:00.094244 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:00.110003 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:00.117285 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:00.148355 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 08:24:00.151898 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:24:00.151937 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:24:00.162294 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240501 08:24:00.304783 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:00.310515 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:00.318027 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:00.324769 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:00.330415 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:24:00.334448 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:00.460906 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:00.460908 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:00.520864 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:00.652966 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:00.659916 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:24:00.694959 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:24:00.721405 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:00.728633 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:00.742102 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:24:00.752493 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:00.752796 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:00.758036 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:24:00.770639 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:00.806905 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:24:00.825906 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:00.869524 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:00.909425 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:00.917783 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:00.936867 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:00.946033 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:00.954782 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:01.101085 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:24:01.138783 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:01.181138 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:01.200554 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:01.236996 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:01.266098 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240501 08:24:01.311362 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:01.318482 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:01.320086 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:01.324447 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:01.379604 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:01.401062 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:01.475775 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:01.490566 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:01.498411 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:01.503954 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:01.514544 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:01.675196 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:24:01.736748 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:01.766456 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:24:01.780323 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:24:01.795492 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:01.808444 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:24:01.882138 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:24:01.915640 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:01.929296 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:01.935117 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:01.941000 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:01.992635 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:02.006577 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:02.059872 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:02.097302 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:24:02.104475 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20240501 08:24:02.121163 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:02.126470 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:02.135627 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:02.163945 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:24:02.183745 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:02.203895 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:24:02.228099 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:02.311764 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240501 08:24:02.318552 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:24:02.394418 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240501 08:24:02.533491 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:02.544080 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:02.562033 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:02.563824 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:02.564074 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:24:02.568640 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:02.585969 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:24:02.587102 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:02.590777 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:24:02.652168 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:02.769872 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:02.778467 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:02.794761 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:02.795136 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:02.804414 27613 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:24:02.804602 27613 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:24:02.805301 27613 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:24:02.814944 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:02.821799 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:24:02.822094 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:24:02.822463 27613 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:24:02.856523 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240501 08:24:02.903147 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:02.928340 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:03.006109 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:03.090620 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:24:03.096232 26363 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:24:03.096401 26363 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:24:03.097016 26363 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:24:03.099170 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:03.099632 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:03.099736 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:24:03.100049 26363 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:24:03.114962 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:03.174103 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:03.180469 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:24:03.193293 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:03.195009 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:24:03.196050 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:03.196875 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:03.202931 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:03.224233 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:24:03.246804 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:03.282668 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:03.299209 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:24:03.366686 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:03.379374 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:03.397668 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:24:03.398394 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:03.398932 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:03.473021 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:03.662807 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:03.674144 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:03.730677 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:03.805629 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:03.811863 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:24:03.815282 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:03.823380 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:03.838989 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:03.991087 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240501 08:24:03.993103 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:24:04.016456 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:04.025709 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:04.042775 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:04.043314 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:04.082839 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:04.092428 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:04.149125 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:04.269109 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:24:04.281191 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:04.312374 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:04.355149 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:24:04.380923 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240501 08:24:04.409854 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:04.415745 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:04.417583 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:04.427317 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:04.431315 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:24:04.499436 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:04.523332 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:04.597327 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:24:04.599617 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:04.619493 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:04.626765 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:04.633352 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
I20240501 08:24:04.638178 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:04.661351 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:04.732383 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:24:04.739857 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240501 08:24:04.746002 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:04.801874 6484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240501 08:24:04.803027 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f090fda814f455a88dccd6680bdd05f: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
W20240501 08:24:04.805413 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:24:04.809156 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:24:04.934824 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240501 08:24:04.938277 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:04.978904 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:24:04.994552 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:05.057873 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:05.065927 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:05.072832 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:05.090282 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:05.159245 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 08:24:05.176923 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:05.253615 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:05.260254 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:05.266917 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:24:05.268040 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:05.285565 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:05.299405 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:05.320324 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:05.357479 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:24:05.447793 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:05.550789 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:05.599414 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:05.599464 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:05.600272 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:05.680287 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
I20240501 08:24:05.692667 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:05.699111 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:05.709440 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:05.710062 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:24:05.711687 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:05.745922 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:05.756182 6702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:24:05.787155 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:05.891045 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:05.894817 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:05.898833 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:05.899778 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:05.921068 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:24:06.003911 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:24:06.048067 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:06.083253 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:06.152817 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:06.193982 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:06.247898 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:06.294392 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:06.300071 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:06.315318 6484 catalog_manager.cc:4573] TS 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d7903106d7e74e1ea102b1abdbfce63e: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:24:06.333395 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:06.350373 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:06.399462 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:06.459645 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240501 08:24:06.476557 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:24:06.506455 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:06.518388 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:06.521813 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:06.539398 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:06.590824 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:06.593276 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:06.679522 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:06.787894 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:06.816689 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:06.871523 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:24:06.900393 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:06.908783 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:06.913749 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:06.914947 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:06.959336 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:24:06.971625 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:24:07.018222 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:07.051054 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:07.057621 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:24:07.101953 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:07.105233 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:07.107839 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:07.119330 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:07.136442 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240501 08:24:07.156399 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:24:07.171077 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:24:07.220191 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:07.225725 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:07.251968 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240501 08:24:07.286152 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:07.300410 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:24:07.325865 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:07.362449 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240501 08:24:07.446513 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:07.514778 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:07.523232 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:07.531142 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:07.553606 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:07.628592 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 08:24:07.629451 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:07.640230 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:24:07.713274 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:07.715543 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:07.726248 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:07.733572 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:07.738056 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:07.811288 6481 catalog_manager.cc:4573] TS a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 193de3c11ae54408a64ae3007bf88920: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:07.851472 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:07.904220 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:08.045907 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:08.055739 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:08.077201 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:08.093900 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:08.113934 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:24:08.128615 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:08.128847 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:08.139756 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:08.144964 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:08.147778 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:08.171736 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:08.200376 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:08.279023 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
I20240501 08:24:08.331470 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:08.333575 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:08.341668 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:08.350895 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:08.366287 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:24:08.391635 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:24:08.427263 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:08.454533 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:08.496556 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:08.672398 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:08.735215 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:08.742604 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:08.751544 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:08.753669 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:08.771899 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:08.816228 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:24:08.922847 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:08.932487 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:08.941015 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:08.946484 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:24:08.958873 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:08.987213 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:08.998592 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:09.093025 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:09.164049 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240501 08:24:09.175042 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:09.201786 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:09.329622 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:09.339669 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:09.342545 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:09.346776 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:09.388250 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 08:24:09.389253 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:09.457716 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:09.462505 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240501 08:24:09.509542 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:09.509900 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:24:09.514413 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:09.518325 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:09.521387 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:09.542078 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:09.570900 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:09.603623 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:24:09.660873 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:09.661651 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:09.742316 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:24:09.795770 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:09.839344 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:09.843036 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:09.860172 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240501 08:24:09.880302 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:24:09.904515 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:09.905895 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:09.920912 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:09.922510 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:09.955279 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:09.987744 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240501 08:24:10.066062 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:24:10.104089 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:10.119626 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:10.122768 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:10.132426 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:10.156464 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:10.167647 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:10.202975 6963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 08:24:10.332404 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:10.374516 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:10.435966 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:10.504016 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:24:10.551972 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:10.557442 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:10.557823 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:10.585129 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:10.595295 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:10.599115 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:10.633152 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:10.699968 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:10.702051 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
I20240501 08:24:10.768813 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:10.776124 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:10.776779 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:10.777050 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:10.788568 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:10.803115 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:10.819195 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:10.889925 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:24:10.947186 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:11.019059 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:11.132680 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:11.160498 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:24:11.169337 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:11.169857 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:11.185331 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:11.188274 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:11.189083 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:11.237188 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:11.347954 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:11.362818 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:11.363199 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:11.373473 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:11.391515 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:11.411520 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:11.450310 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240501 08:24:11.499974 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:11.503912 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:11.515985 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:11.558092 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:11.713304 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:11.767414 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:11.781704 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:11.784400 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:11.797286 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:11.836594 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:11.858440 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:11.888942 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240501 08:24:11.906340 28885 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:24:11.906535 28885 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:24:11.907238 28885 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:24:11.910315 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:11.910514 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:11.910646 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:24:11.910918 28885 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:24:11.950294 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:11.957774 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:11.969630 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:11.983178 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:12.010371 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:12.086149 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:12.120226 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:12.130105 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:12.149618 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:12.155953 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:24:12.189388 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:12.310281 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:12.333699 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:12.376379 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:24:12.392887 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:12.399369 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:12.411895 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:12.417136 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:12.436410 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:12.488098 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:12.518328 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:12.545055 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:12.623621 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
I20240501 08:24:12.627683 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:12.631335 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:12.633805 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:12.642549 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:12.726622 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:12.727818 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240501 08:24:12.748432 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:12.780668 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 08:24:12.781450 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:12.834115 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:12.953091 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:12.990211 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:13.020200 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:13.042130 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:13.051052 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:13.056715 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:13.158491 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:13.204012 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
I20240501 08:24:13.229135 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:13.243348 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:13.253846 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:13.261163 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:13.270912 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:13.305385 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:13.331938 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:13.335880 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:13.372905 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:13.424557 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:13.484730 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:13.520130 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:13.599252 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:13.657238 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:24:13.662762 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:13.668160 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:13.668551 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:13.670118 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:13.765560 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:13.827651 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:13.853056 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:13.858717 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:13.867779 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:13.871737 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:13.874770 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:24:13.877609 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:13.912753 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:13.920688 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:14.055361 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
I20240501 08:24:14.248494 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:14.256261 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:14.268041 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:14.273799 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:24:14.278542 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:14.310047 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:14.317160 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:14.318727 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:14.333424 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:14.339594 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:14.343780 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:24:14.424189 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:14.446406 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:14.456627 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:14.457077 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:14.461602 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240501 08:24:14.478950 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 08:24:14.663033 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:14.754575 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:14.784410 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:14.791646 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:24:14.796525 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:14.812582 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:14.846753 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:14.856881 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:24:14.865658 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:14.875069 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:14.878787 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:24:14.883344 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:14.887452 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:14.898473 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:14.917464 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:14.922446 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:15.054361 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240501 08:24:15.071293 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:15.073856 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:15.075012 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:15.077577 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:15.089358 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240501 08:24:15.093344 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:15.223899 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 08:24:15.287927 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:15.329210 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:15.420071 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:15.455955 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:15.462401 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:15.486510 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:15.490286 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:15.508541 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:15.540282 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240501 08:24:15.566236 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:15.597417 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:24:15.671900 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:15.676043 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:15.691342 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:15.708006 29380 raft_consensus.cc:482] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19fdb76c8d1f41d7842e397adfc1ef3f)
I20240501 08:24:15.708158 29380 raft_consensus.cc:504] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "19fdb76c8d1f41d7842e397adfc1ef3f" member_type: VOTER last_known_addr { host: "127.25.107.8" port: 41625 } } peers { permanent_uuid: "aba7c226aaca4e8c90931f547d48aaa3" member_type: VOTER last_known_addr { host: "127.25.107.6" port: 41219 } } peers { permanent_uuid: "e3a1c4a5b44340be8a183d5d649b27ac" member_type: VOTER last_known_addr { host: "127.25.107.3" port: 43505 } }
I20240501 08:24:15.708729 29380 leader_election.cc:288] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625), aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219)
W20240501 08:24:15.710284 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:15.710459 6833 leader_election.cc:334] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:15.710534 6833 leader_election.cc:302] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3a1c4a5b44340be8a183d5d649b27ac; no voters: 19fdb76c8d1f41d7842e397adfc1ef3f, aba7c226aaca4e8c90931f547d48aaa3
I20240501 08:24:15.710723 29380 raft_consensus.cc:2726] T 049090443eaa4b18ba90cb60af9a87ab P e3a1c4a5b44340be8a183d5d649b27ac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:24:15.712132 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:15.798269 29411 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:24:15.798446 29411 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:24:15.799036 29411 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:24:15.800236 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:15.800375 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:24:15.800459 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:24:15.800683 29411 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:24:15.853886 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:15.870203 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:15.929453 6703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:24:15.948029 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:15.984220 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:24:16.009668 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:16.099454 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:16.103165 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:24:16.117364 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:16.118693 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:16.122227 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:24:16.124579 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:16.161159 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:16.179925 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:16.186127 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:24:16.321655 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:16.327702 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:16.332127 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:16.353034 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:16.353111 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:16.363229 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:16.387246 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:16.404582 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:16.455832 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240501 08:24:16.479139 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:16.621960 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:24:16.738394 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:16.747639 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:16.756246 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:16.769225 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:16.804425 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:16.815183 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:16.885907 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240501 08:24:16.888967 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:16.913465 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:16.918124 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:16.920847 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:16.930544 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:16.936249 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:24:16.961308 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:16.967255 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:16.972396 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:17.077318 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:24:17.179245 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:17.226334 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:24:17.325747 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:17.341619 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:17.346089 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:24:17.347723 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:17.348234 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:17.352083 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:17.373454 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:24:17.379968 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:17.422606 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:17.438563 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240501 08:24:17.442091 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:24:17.450788 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:24:17.488464 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:17.507550 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240501 08:24:17.526684 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:17.528872 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:17.547197 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:17.549329 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:17.711841 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:17.817631 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 08:24:17.850857 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:17.904156 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:17.909068 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:24:17.970448 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:17.981523 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:17.993283 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:18.009753 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:18.017992 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:18.152053 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:18.165238 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:24:18.189934 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240501 08:24:18.192219 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:18.199821 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:18.205328 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:18.207866 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:18.228731 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:18.285632 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:24:18.409606 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:18.463654 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:18.528105 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:18.610152 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:18.628401 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:18.628931 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:18.631660 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:24:18.635113 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:18.638747 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:18.652796 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:24:18.681944 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:24:18.763975 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:24:18.850569 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:24:18.857784 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:18.857894 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:24:18.859742 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:18.866079 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:18.869772 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:19.020267 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:19.021301 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:19.097702 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:19.138152 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:19.160809 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240501 08:24:19.194620 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:19.209280 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240501 08:24:19.261258 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:19.265995 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:19.276924 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:19.277338 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:19.421192 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:19.423926 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:24:19.431074 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:19.446467 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:19.461390 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:19.461825 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:19.478580 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:24:19.496949 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:19.560923 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:19.563992 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240501 08:24:19.635478 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:24:19.645126 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:24:19.654919 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:24:19.670984 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240501 08:24:19.735531 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:19.739568 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:19.799774 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:19.805269 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:19.806936 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:24:19.835327 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:19.835894 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:19.848454 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:19.850349 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:19.860865 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:19.861229 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:20.008345 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
I20240501 08:24:20.047638 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:20.047747 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:20.055119 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:20.074301 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:20.197331 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:20.202593 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:24:20.216635 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:20.292681 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:20.317612 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240501 08:24:20.339972 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:20.374984 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111) [suppressed 111 similar messages]
I20240501 08:24:20.454810 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:20.456606 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:20.469645 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:20.484048 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:20.601713 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:24:20.649732 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:20.661091 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:20.662511 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:20.667680 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:20.740736 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:20.782332 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:20.790438 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:24:20.820516 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:20.893301 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:20.945457 6705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 08:24:21.003298 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:24:21.012305 30145 raft_consensus.cc:482] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a518ca5e9de24204bd028b0ce4a77d58)
I20240501 08:24:21.012476 30145 raft_consensus.cc:504] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd3434d87b3c46c6848a711cdf6b150c" member_type: VOTER last_known_addr { host: "127.25.107.2" port: 46507 } } peers { permanent_uuid: "a518ca5e9de24204bd028b0ce4a77d58" member_type: VOTER last_known_addr { host: "127.25.107.7" port: 33929 } } peers { permanent_uuid: "0a814bba1abe4653bcd418a83ec42370" member_type: VOTER last_known_addr { host: "127.25.107.5" port: 45691 } }
I20240501 08:24:21.013164 30145 leader_election.cc:288] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929), 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691)
W20240501 08:24:21.014495 6702 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:21.015286 6705 leader_election.cc:334] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111)
I20240501 08:24:21.015390 6705 leader_election.cc:302] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3434d87b3c46c6848a711cdf6b150c; no voters: 0a814bba1abe4653bcd418a83ec42370, a518ca5e9de24204bd028b0ce4a77d58
I20240501 08:24:21.017635 30145 raft_consensus.cc:2726] T 070e927b5d684367ae0b40b12a23da54 P cd3434d87b3c46c6848a711cdf6b150c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:24:21.072731 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:24:21.084950 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:21.093341 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:21.094098 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:21.098515 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:21.106621 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:24:21.112797 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:21.201970 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:21.238049 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:24:21.253204 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:24:21.262528 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:21.273481 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:21.273882 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:21.274185 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:21.278861 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:21.291674 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:21.386487 6703 consensus_peers.cc:485] T a7a56c284444496b8579600a49f19d1d P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:21.430552 6962 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:24:21.480993 6705 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240501 08:24:21.582712 6963 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:24:21.594986 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
I20240501 08:24:21.679126 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:21.680415 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:21.697710 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:21.701030 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:21.718384 6833 consensus_peers.cc:485] T d7903106d7e74e1ea102b1abdbfce63e P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:21.779969 6962 consensus_peers.cc:485] T dc19d80bad46460087c143778a4b0638 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240501 08:24:21.795434 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:24:21.847236 6963 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:21.865928 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:21.868472 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:21.892568 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:21.900946 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:21.915200 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:21.932088 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:24:21.941918 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:21.962456 6572 consensus_peers.cc:485] T 29506d751571401f86bced7e6f5415d7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:24:22.014688 6963 consensus_peers.cc:485] T a25e1ddb987b4f8c8a4031053f581c7c P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:24:22.024135 6705 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 0a814bba1abe4653bcd418a83ec42370 (127.25.107.5:45691): Couldn't send request to peer 0a814bba1abe4653bcd418a83ec42370. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.5:45691: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240501 08:24:22.061069 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:22.075069 6833 consensus_peers.cc:485] T 399d1a392b3f4d9e8531b6e0b5ad2a61 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:22.131331 6703 consensus_peers.cc:485] T 8b7f87b6032d45c5be5d748ab2cc71e2 P cd3434d87b3c46c6848a711cdf6b150c -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:24:22.265987 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:22.312395 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:22.324469 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:22.330034 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:24:22.336341 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:22.337574 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:22.403313 6963 consensus_peers.cc:485] T e9629fbbb4554edc9ba341d42b51f252 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:22.408574 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:22.413787 6572 consensus_peers.cc:485] T 491674dfbf74498e8a787bebaa5e95da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:22.452229 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:22.455339 6702 consensus_peers.cc:485] T fdaad8e1d86942ddbc97958dcf31b344 P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20240501 08:24:22.527619 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:22.539512 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:22.546694 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:22.566141 6833 consensus_peers.cc:485] T cfc0c7b429754d92b4dd166c073ac8ed P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:24:22.566972 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:24:22.689522 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:22.754108 6963 consensus_peers.cc:485] T 5f090fda814f455a88dccd6680bdd05f P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:24:22.762331 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:22.917035 6833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:24:22.919785 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:22.933447 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
I20240501 08:24:22.944828 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:22.946175 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:22.973102 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:22.986220 6572 consensus_peers.cc:485] T 13b4e2d5fe4846e8b3c6081abb3c53da P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:24:22.992444 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:22.994279 6572 consensus_peers.cc:485] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:24:22.997143 6702 consensus_peers.cc:485] T f93c2d0f9c7947bb83a50c7fb8c3518a P cd3434d87b3c46c6848a711cdf6b150c -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20240501 08:24:23.168578 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:23.184502 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:23.185132 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:23.198771 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:23.321337 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
W20240501 08:24:23.410166 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:23.442035 6833 consensus_peers.cc:485] T 658f2a172a2d47c8bf8b44c9d51132e4 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:23.486004 6963 consensus_peers.cc:485] T 4a6efa669c1645aaa971d59e5d38c8d5 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Couldn't send request to peer 19fdb76c8d1f41d7842e397adfc1ef3f. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:24:23.501004 6833 consensus_peers.cc:485] T 6575b94bd51546f1bd3bb862f012e7ca P e3a1c4a5b44340be8a183d5d649b27ac -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:24:23.513523 6833 consensus_peers.cc:462] T 30b4f075057e42d0bb3f8cdabdfbc643 P e3a1c4a5b44340be8a183d5d649b27ac -> Peer 19fdb76c8d1f41d7842e397adfc1ef3f (127.25.107.8:41625): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.8:41625: connect: Connection refused (error 111)
W20240501 08:24:23.553572 6963 consensus_peers.cc:485] T 60238162349647c681685349c6f8ede4 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:24:23.594583 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:23.605732 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:23.612325 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:24:23.627679 6572 consensus_peers.cc:485] T 37a6157d7d734a80900139944481928c P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn't send request to peer aba7c226aaca4e8c90931f547d48aaa3. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:24:23.629380 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:23.718847 6962 consensus_peers.cc:485] T a9e70954aa4e4fcfbad12e8bfe3e0059 P 28b7613bd4834f3ab245c82dd7e80240 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Couldn't send request to peer a518ca5e9de24204bd028b0ce4a77d58. Status: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:24:23.749361 6571 consensus_peers.cc:462] T 358b6fe9c68349cb8cb963bb3a9ed2e9 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer a518ca5e9de24204bd028b0ce4a77d58 (127.25.107.7:33929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.7:33929: connect: Connection refused (error 111)
I20240501 08:24:23.788336 7008 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:23.802897 6618 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:24:23.809542 6748 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:24:23.825717 6877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:24:23.918171 6572 consensus_peers.cc:462] T 4db7f887af8f425c8fa661196add4d37 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.107.6:41219: connect: Connection refused (error 111)
W20240501 08:24:23.963423 6572 consensus_peers.cc:485] T 9b9bd7500e8747c6b8ee94a1faa1c5f7 P 97cdee0bf9d84645a1451f9822c5d699 -> Peer aba7c226aaca4e8c90931f547d48aaa3 (127.25.107.6:41219): Couldn'